+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit
+ bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190926/B1b --output-run DEEP/20190926/B1b/science#step0/20240311T225937Z --qgraph pipeline.qgraph
lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process.
lsst.ctrl.bps.drivers INFO: Starting submission process
lsst.ctrl.bps.drivers INFO: Initializing execution environment
lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 9.7252 seconds; current memory usage: 0.180 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph)
lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.6648 seconds
lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.6957 seconds
lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/pipeline_orig.qgraph DEEP/20190926/B1b/science#step0/20240311T225937Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/pipeline.qgraph 
lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.6227 seconds
lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 10.3567 seconds; current memory usage: 0.276 Gibyte, delta: 0.096 Gibyte, peak delta: 0.084 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.276 Gibyte (main), 0.267 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs)
lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0122 seconds; current memory usage: 0.276 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.276 Gibyte (main), 0.267 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 372 cluster(s)
lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow)
lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190926_B1b_science#step0_20240311T225937Z'
lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.7863 seconds; current memory usage: 0.276 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.276 Gibyte (main), 0.267 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 374 job(s) (including final)
lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow)
parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses
    s_addresses.add(address_by_interface(interface))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface
    return socket.inet_ntoa(fcntl.ioctl(
                            ^^^^^^^^^^^^
OSError: [Errno 99] Cannot assign requested address
lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z
lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 2.1375 seconds; current memory usage: 0.277 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.277 Gibyte (main), 0.267 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting submit stage
lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution
parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000
parsl.dataflow.dflow INFO: Starting DataFlowKernel with config
Config(
    app_cache=True, 
    checkpoint_files=None, 
    checkpoint_mode='task_exit', 
    checkpoint_period=None, 
    executors=[MultiHighThroughputExecutor()], 
    garbage_collect=True, 
    initialize_logging=True, 
    internal_tasks_max_threads=10, 
    max_idletime=120.0, 
    monitoring=None, 
    retries=1, 
    retry_handler=None, 
    run_dir='runinfo', 
    strategy='simple', 
    usage_tracking=False
)
parsl.dataflow.dflow INFO: Parsl version: 2023.06.12
parsl.usage_tracking.usage DEBUG: Tracking status: False
parsl.dataflow.dflow INFO: Run id is: 6fcbfb49-5d6e-425b-bbb5-43e3ab5a20c0
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py
parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name
parsl.dataflow.memoization INFO: App caching initialized
parsl.dataflow.strategy DEBUG: Scaling strategy: simple
parsl.executors.high_throughput.executor DEBUG: Starting queue management thread
parsl.executors.high_throughput.executor DEBUG: queue management worker starting
parsl.executors.high_throughput.executor DEBUG: Started queue management thread
Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.executors.high_throughput.executor DEBUG: Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 23247675815680)>
parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider:
<proc_lsst.multi.MultiProvider object at 0x1524c6b61f10>
parsl.executors.status_handling INFO: Scaling out by 1 blocks
parsl.executors.status_handling INFO: Allocated block ID 0
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0
2024-03-11 23:00:25 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
2024-03-11 23:00:25 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-11 23:00:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:25 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
parsl.providers.local.local DEBUG: Launching in remote mode
2024-03-11 23:00:26 proc_lsst.multi:170 [INFO]  [multi] job_id 26021
proc_lsst.multi INFO: [multi] job_id 26021
2024-03-11 23:00:26 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:26 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:26 proc_lsst.multi:178 [INFO]  [multi] provider local accepted submit and returned 26021
proc_lsst.multi INFO: [multi] provider local accepted submit and returned 26021
parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 26021
parsl.dataflow.job_status_poller DEBUG: Adding executor multi
parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: <AppFuture at 0x1524c6a9a090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 11ca421e45978ad36a0ed58c6fe36072
parsl.dataflow.memoization INFO: Task 0 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a093a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d1f8bf7b-6764-4331-913b-b056979dd644_isrForCrosstalkSources_898387_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d1f8bf7b-6764-4331-913b-b056979dd644_isrForCrosstalkSources_898387_44.stderr
parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: <AppFuture at 0x1524d14fa110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1 has memoization hash a5ed114523d75c7fdd1a4b55f8da638d
parsl.dataflow.memoization INFO: Task 1 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a096c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/87ff1644-1a9a-463b-8fbb-db1aa991d608_isrForCrosstalkSources_898282_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/87ff1644-1a9a-463b-8fbb-db1aa991d608_isrForCrosstalkSources_898282_46.stderr
parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: <AppFuture at 0x1524c6d6f950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 5ce01a0f9ad153380448a357028d266d
parsl.dataflow.memoization INFO: Task 2 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a099e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2dc7cf98-e7d2-4d09-b3cc-7cdf8ec7efb0_isrForCrosstalkSources_898281_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2dc7cf98-e7d2-4d09-b3cc-7cdf8ec7efb0_isrForCrosstalkSources_898281_42.stderr
parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: <AppFuture at 0x1524c6d6fa50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 9cd0b5350c2d4ed1e7b75042759ebfe3
parsl.dataflow.memoization INFO: Task 3 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a098a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/120e993b-91cd-4a61-a610-ccbd8db53cfa_isrForCrosstalkSources_898280_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/120e993b-91cd-4a61-a610-ccbd8db53cfa_isrForCrosstalkSources_898280_12.stderr
parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: <AppFuture at 0x1524d12dc310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 086dc1ccf9ca50359f02f734ed98a834
parsl.dataflow.memoization INFO: Task 4 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a09b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d4e3616b-dedb-4c8d-ae1c-1504d7d33b77_isrForCrosstalkSources_898282_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d4e3616b-dedb-4c8d-ae1c-1504d7d33b77_isrForCrosstalkSources_898282_33.stderr
parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: <AppFuture at 0x1524c6d69650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 35fc7be801d17c7572222c4dcc43ee99
parsl.dataflow.memoization INFO: Task 5 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a09940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/f4835ebb-2d42-4322-90e1-c40d75273ecc_isrForCrosstalkSources_898385_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/f4835ebb-2d42-4322-90e1-c40d75273ecc_isrForCrosstalkSources_898385_31.stderr
parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: <AppFuture at 0x1524c6bf4050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 52c7b28b0ab4f9b9f0a6818a83bd81bf
parsl.dataflow.memoization INFO: Task 6 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a09bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c16a69ca-a9bc-464b-b1b8-484f8ff75d9b_isrForCrosstalkSources_898280_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c16a69ca-a9bc-464b-b1b8-484f8ff75d9b_isrForCrosstalkSources_898280_25.stderr
parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: <AppFuture at 0x1524d133edd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 3717f5d8abe58acbb4e537f69297e374
parsl.dataflow.memoization INFO: Task 7 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a09260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/f18d04ab-b8f6-4f57-ac6c-55ffcbcddd32_isrForCrosstalkSources_898282_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/f18d04ab-b8f6-4f57-ac6c-55ffcbcddd32_isrForCrosstalkSources_898282_38.stderr
parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: <AppFuture at 0x1524c6d7c890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 3d3c55233080a8f9be7a787ba09f7793
parsl.dataflow.memoization INFO: Task 8 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a09da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/cae1c742-25b8-4a80-af3e-2e635e190bf3_isrForCrosstalkSources_898282_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/cae1c742-25b8-4a80-af3e-2e635e190bf3_isrForCrosstalkSources_898282_21.stderr
parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: <AppFuture at 0x1524c6d5b110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 80a7f00a8fc879649bd4744a2b027556
parsl.dataflow.memoization INFO: Task 9 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a09580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/a3da6af5-cd37-47ba-9ebb-98041c71aa30_isrForCrosstalkSources_898281_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/a3da6af5-cd37-47ba-9ebb-98041c71aa30_isrForCrosstalkSources_898281_59.stderr
parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: <AppFuture at 0x1524c6d308d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 513ff602bc17cee8c13455908fe437e6
parsl.dataflow.memoization INFO: Task 10 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a09a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/dee4bbee-c004-49df-bd85-9f67f273b108_isrForCrosstalkSources_898385_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/dee4bbee-c004-49df-bd85-9f67f273b108_isrForCrosstalkSources_898385_57.stderr
parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: <AppFuture at 0x1524c6d33b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 420e054faea92458bc3d5b9cc6aa6f11
parsl.dataflow.memoization INFO: Task 11 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a09d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/346cc58c-809c-447b-8ef3-291001432c58_isrForCrosstalkSources_898280_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/346cc58c-809c-447b-8ef3-291001432c58_isrForCrosstalkSources_898280_9.stderr
parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: <AppFuture at 0x1524c6d12c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 6c69d94f8c403a3a03c2b56d024a9264
parsl.dataflow.memoization INFO: Task 12 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/6ac9ddaf-f15c-4a50-b9a4-bcc11d6695a6_isrForCrosstalkSources_898386_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/6ac9ddaf-f15c-4a50-b9a4-bcc11d6695a6_isrForCrosstalkSources_898386_25.stderr
parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: <AppFuture at 0x1524c6d12110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 13 has memoization hash d3ae44d160f0fce202da2e3e9d8917e7
parsl.dataflow.memoization INFO: Task 13 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2ff55ecb-b7b2-495a-9b4d-576033e50d67_isrForCrosstalkSources_898282_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2ff55ecb-b7b2-495a-9b4d-576033e50d67_isrForCrosstalkSources_898282_6.stderr
parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: <AppFuture at 0x1524c6d1ed90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 14 has memoization hash e0032e36910d48b9f87b793edd9c490a
parsl.dataflow.memoization INFO: Task 14 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7753cdbd-048d-4206-af67-96435b5587f9_isrForCrosstalkSources_898385_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7753cdbd-048d-4206-af67-96435b5587f9_isrForCrosstalkSources_898385_56.stderr
parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: <AppFuture at 0x1524c6d1df90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 2b5f5c5455e1ee9cb7e81278f05611a1
parsl.dataflow.memoization INFO: Task 15 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a09f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/dcc1328a-8406-471f-a410-092227f93719_isrForCrosstalkSources_898281_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/dcc1328a-8406-471f-a410-092227f93719_isrForCrosstalkSources_898281_13.stderr
parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: <AppFuture at 0x1524c6b913d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 885a563d12b8c4f1fde90c52c4b7f8a7
parsl.dataflow.memoization INFO: Task 16 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/17eec91b-d77f-4afd-8eeb-0d6181b36860_isrForCrosstalkSources_898280_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/17eec91b-d77f-4afd-8eeb-0d6181b36860_isrForCrosstalkSources_898280_13.stderr
parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: <AppFuture at 0x1524c6d58350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 87ba32e8034a25bd2d7307e410f9e355
parsl.dataflow.memoization INFO: Task 17 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/573546e3-2299-40f5-aef8-894eb4975890_isrForCrosstalkSources_898385_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/573546e3-2299-40f5-aef8-894eb4975890_isrForCrosstalkSources_898385_3.stderr
parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: <AppFuture at 0x1524c6d00e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 18 has memoization hash ed298d258ec787a7215ba929fce85e8a
parsl.dataflow.memoization INFO: Task 18 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/eb30a0c0-81ee-40ad-a344-19d1c6e832b2_isrForCrosstalkSources_898282_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/eb30a0c0-81ee-40ad-a344-19d1c6e832b2_isrForCrosstalkSources_898282_51.stderr
parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: <AppFuture at 0x1524c6d2b010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 19 has memoization hash a4ae9fbcf3cfc6296244e4fe3eef11ef
parsl.dataflow.memoization INFO: Task 19 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b86a6793-05f9-48b9-aebe-0be1b346d28c_isrForCrosstalkSources_898282_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b86a6793-05f9-48b9-aebe-0be1b346d28c_isrForCrosstalkSources_898282_24.stderr
parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: <AppFuture at 0x1524c6d2a490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 20 has memoization hash fb4bf52421a254f1d7e5ab2272ab0a48
parsl.dataflow.memoization INFO: Task 20 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/bead777e-069e-4dd5-96e0-35923c5bd1b3_isrForCrosstalkSources_898385_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/bead777e-069e-4dd5-96e0-35923c5bd1b3_isrForCrosstalkSources_898385_24.stderr
parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: <AppFuture at 0x1524c6cf6d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 21 has memoization hash eaba84f095013a01360f6f57ea021b53
parsl.dataflow.memoization INFO: Task 21 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/d391c3ef-c633-4039-b849-f78ab141ee93_isrForCrosstalkSources_898281_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/d391c3ef-c633-4039-b849-f78ab141ee93_isrForCrosstalkSources_898281_16.stderr
parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: <AppFuture at 0x1524c6cf6450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 431fbec67b1e77161ff8aa712991f01a
parsl.dataflow.memoization INFO: Task 22 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/6e950a10-9a4f-420f-9ff3-044ea88f8804_isrForCrosstalkSources_898386_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/6e950a10-9a4f-420f-9ff3-044ea88f8804_isrForCrosstalkSources_898386_29.stderr
parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: <AppFuture at 0x1524c6ceda10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 23 has memoization hash d1195ad3eb3a2fcedcee37148a66dd04
parsl.dataflow.memoization INFO: Task 23 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4cfe8fd0-2867-4f4c-8179-738f586e8e26_isrForCrosstalkSources_898282_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4cfe8fd0-2867-4f4c-8179-738f586e8e26_isrForCrosstalkSources_898282_61.stderr
parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: <AppFuture at 0x1524c6cef010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 5387d8e16a6fccd8f2900e215541411e
parsl.dataflow.memoization INFO: Task 24 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/79b75c9d-077c-468c-8338-7205a6c9c41a_isrForCrosstalkSources_898386_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/79b75c9d-077c-468c-8338-7205a6c9c41a_isrForCrosstalkSources_898386_35.stderr
parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: <AppFuture at 0x1524c6cdaa50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 25 has memoization hash fb974b2aac315105cb406022bf0916c8
parsl.dataflow.memoization INFO: Task 25 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/6caeb0fc-eedc-4e34-96b0-9a6aea420ee0_isrForCrosstalkSources_898387_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/6caeb0fc-eedc-4e34-96b0-9a6aea420ee0_isrForCrosstalkSources_898387_58.stderr
parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: <AppFuture at 0x1524c6cda4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 26 has memoization hash f66613753d89e86d88dbbc739363e151
parsl.dataflow.memoization INFO: Task 26 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b9f828d8-a2d0-40f9-8610-b9d329bf30a9_isrForCrosstalkSources_898282_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b9f828d8-a2d0-40f9-8610-b9d329bf30a9_isrForCrosstalkSources_898282_17.stderr
parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: <AppFuture at 0x1524c6cda290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 618cd96555ea1e68f17ff6c1566fe106
parsl.dataflow.memoization INFO: Task 27 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/321865f4-8612-464c-a481-8f132f4c263a_isrForCrosstalkSources_898281_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/321865f4-8612-464c-a481-8f132f4c263a_isrForCrosstalkSources_898281_43.stderr
parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: <AppFuture at 0x1524c6cce4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 28 has memoization hash dcc410a4a01ae0a5137e4ca02474af01
parsl.dataflow.memoization INFO: Task 28 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d88b873d-cb41-4024-8ee3-2f5fe70f4756_isrForCrosstalkSources_898387_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d88b873d-cb41-4024-8ee3-2f5fe70f4756_isrForCrosstalkSources_898387_7.stderr
parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: <AppFuture at 0x1524c6ccfc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 29 has memoization hash a6f7cae6b73ffe97519f9b27d206174c
parsl.dataflow.memoization INFO: Task 29 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bc40e429-4e5c-460e-9d70-35065e086810_isrForCrosstalkSources_898282_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bc40e429-4e5c-460e-9d70-35065e086810_isrForCrosstalkSources_898282_29.stderr
parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: <AppFuture at 0x1524c6d11d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 30 has memoization hash f58fb1a4d2691383d4fd81852bdce24b
parsl.dataflow.memoization INFO: Task 30 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/6917180e-6c4a-4126-bed4-fc5f56134766_isrForCrosstalkSources_898282_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/6917180e-6c4a-4126-bed4-fc5f56134766_isrForCrosstalkSources_898282_22.stderr
parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: <AppFuture at 0x1524c6cc0110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 31 has memoization hash ba5315b8ce456764b60508be2d48bfc8
parsl.dataflow.memoization INFO: Task 31 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/039b7c8f-1cb2-43f2-a38e-67a6ad248127_isrForCrosstalkSources_898387_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/039b7c8f-1cb2-43f2-a38e-67a6ad248127_isrForCrosstalkSources_898387_52.stderr
parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: <AppFuture at 0x1524c6cc0250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 2c4b109187beca62538e07bf6968b7ab
parsl.dataflow.memoization INFO: Task 32 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/635a7426-b3ad-4662-882f-a61e98e19d94_isrForCrosstalkSources_898387_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/635a7426-b3ad-4662-882f-a61e98e19d94_isrForCrosstalkSources_898387_4.stderr
parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: <AppFuture at 0x1524c6d13590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 33 has memoization hash aebfbd9cd1d96f6c0099067496c5ea61
parsl.dataflow.memoization INFO: Task 33 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/8caf94f5-495c-4b77-aaaa-6f735d7d7ce2_isrForCrosstalkSources_898282_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/8caf94f5-495c-4b77-aaaa-6f735d7d7ce2_isrForCrosstalkSources_898282_36.stderr
parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: <AppFuture at 0x1524c6cba4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 17b9893eea0e427ef7400a1c09060559
parsl.dataflow.memoization INFO: Task 34 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/402c66fc-737b-46e0-914e-a26375b729af_isrForCrosstalkSources_898386_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/402c66fc-737b-46e0-914e-a26375b729af_isrForCrosstalkSources_898386_46.stderr
parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: <AppFuture at 0x1524c6cb90d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 35 has memoization hash c3092d181b97d4faadb9921f7355fedc
parsl.dataflow.memoization INFO: Task 35 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/f3dd7e49-7940-4eeb-8cba-b244236d7937_isrForCrosstalkSources_898387_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/f3dd7e49-7940-4eeb-8cba-b244236d7937_isrForCrosstalkSources_898387_9.stderr
parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: <AppFuture at 0x1524c6cb8750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 3f9632ca4dae868d4f01bace07d1c105
parsl.dataflow.memoization INFO: Task 36 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/fc75db1d-355f-4032-a303-e530543d4707_isrForCrosstalkSources_898280_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/fc75db1d-355f-4032-a303-e530543d4707_isrForCrosstalkSources_898280_38.stderr
parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: <AppFuture at 0x1524c6ca4490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 1bdaf16dbac79324094dbf4419a5bea7
parsl.dataflow.memoization INFO: Task 37 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/15bdc7f3-9bc4-443b-bae4-afdf02ee76fe_isrForCrosstalkSources_898281_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/15bdc7f3-9bc4-443b-bae4-afdf02ee76fe_isrForCrosstalkSources_898281_3.stderr
parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: <AppFuture at 0x1524c6cbb810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 46c13619215e51d75e926ac67187d5d6
parsl.dataflow.memoization INFO: Task 38 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bd138c97-a6c3-4744-bc05-c8814b914e94_isrForCrosstalkSources_898281_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bd138c97-a6c3-4744-bc05-c8814b914e94_isrForCrosstalkSources_898281_60.stderr
parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: <AppFuture at 0x1524c6ca27d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 39 has memoization hash ef7a45a96f7383ffb5525e18b549ae55
parsl.dataflow.memoization INFO: Task 39 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/60a5a6d5-c04b-49d8-8129-a4ab43e0a21d_isrForCrosstalkSources_898282_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/60a5a6d5-c04b-49d8-8129-a4ab43e0a21d_isrForCrosstalkSources_898282_48.stderr
parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: <AppFuture at 0x1524c6ca18d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 1d3f9959c15372d9cc0c8de8f9dd8e0d
parsl.dataflow.memoization INFO: Task 40 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/5b444700-cb54-40e5-b3f3-52df420641b9_isrForCrosstalkSources_898386_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/5b444700-cb54-40e5-b3f3-52df420641b9_isrForCrosstalkSources_898386_7.stderr
parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: <AppFuture at 0x1524c6ca3890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 74c83a57394281c52e550eb0bdd2fcb3
parsl.dataflow.memoization INFO: Task 41 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/abf88f43-a134-4870-b03f-8bf50c98055c_isrForCrosstalkSources_898386_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/abf88f43-a134-4870-b03f-8bf50c98055c_isrForCrosstalkSources_898386_61.stderr
parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: <AppFuture at 0x1524c6c95d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 42 has memoization hash de3ec7bda6305a18308a01ee01bb2254
parsl.dataflow.memoization INFO: Task 42 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/826c11e3-7b71-4bce-9741-eae5eed04a7a_isrForCrosstalkSources_898386_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/826c11e3-7b71-4bce-9741-eae5eed04a7a_isrForCrosstalkSources_898386_4.stderr
parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: <AppFuture at 0x1524c6c97290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 43 has memoization hash adab158eadf1f429f9d20c17f4b45a83
parsl.dataflow.memoization INFO: Task 43 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/dfe186ea-5d2b-4b7f-aeb5-8b2d93546cdc_isrForCrosstalkSources_898280_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/dfe186ea-5d2b-4b7f-aeb5-8b2d93546cdc_isrForCrosstalkSources_898280_16.stderr
parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: <AppFuture at 0x1524c6cec5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 7acdac1e3505a316c1da629bd992d66e
parsl.dataflow.memoization INFO: Task 44 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/0da73840-5d80-4d93-9e41-6b4cc2432474_isrForCrosstalkSources_898280_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/0da73840-5d80-4d93-9e41-6b4cc2432474_isrForCrosstalkSources_898280_29.stderr
parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: <AppFuture at 0x1524c6c87c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 91745207ed642774a743cd90f56b71d1
parsl.dataflow.memoization INFO: Task 45 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/759c3fc2-e728-4d6a-adff-d78ffabc95a3_isrForCrosstalkSources_898387_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/759c3fc2-e728-4d6a-adff-d78ffabc95a3_isrForCrosstalkSources_898387_21.stderr
parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: <AppFuture at 0x1524c6c85650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 18a2b4d4c48b2d0a9643b9828bf24975
parsl.dataflow.memoization INFO: Task 46 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/d5d0b886-8bd8-4376-a37f-5f7a7ce9f534_isrForCrosstalkSources_898280_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/d5d0b886-8bd8-4376-a37f-5f7a7ce9f534_isrForCrosstalkSources_898280_22.stderr
parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: <AppFuture at 0x1524c6c86910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 7b62488298adb43db8b17fbff1ae1a27
parsl.dataflow.memoization INFO: Task 47 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c79f167f-8e27-41d0-a80e-58de9a33826a_isrForCrosstalkSources_898282_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c79f167f-8e27-41d0-a80e-58de9a33826a_isrForCrosstalkSources_898282_41.stderr
parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: <AppFuture at 0x1524c6e7ead0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 48 has memoization hash f39462653ace5ab3806b8479dbda6085
parsl.dataflow.memoization INFO: Task 48 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/ec3c5e95-bf39-4440-b2fd-3f3c11fb36ea_isrForCrosstalkSources_898282_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/ec3c5e95-bf39-4440-b2fd-3f3c11fb36ea_isrForCrosstalkSources_898282_23.stderr
parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: <AppFuture at 0x1524c6e7cdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 49 has memoization hash ad97f3914a37e5b02af130822ba11a7e
parsl.dataflow.memoization INFO: Task 49 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/27bc8482-4453-4fbf-99b1-4f3a673b4b28_isrForCrosstalkSources_898386_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/27bc8482-4453-4fbf-99b1-4f3a673b4b28_isrForCrosstalkSources_898386_32.stderr
parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: <AppFuture at 0x1524c6e6e0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 50 has memoization hash c6ba26ba3d612f7c1d4a0c862e306776
parsl.dataflow.memoization INFO: Task 50 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/08feae9c-1183-4ffe-ac12-6b5e366c193b_isrForCrosstalkSources_898386_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/08feae9c-1183-4ffe-ac12-6b5e366c193b_isrForCrosstalkSources_898386_26.stderr
parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: <AppFuture at 0x1524c6e6fcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 7546d6afece851de1a9a849df089a099
parsl.dataflow.memoization INFO: Task 51 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/9ca8a2b7-4ecb-4039-b5e8-82827328df46_isrForCrosstalkSources_898282_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/9ca8a2b7-4ecb-4039-b5e8-82827328df46_isrForCrosstalkSources_898282_40.stderr
parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: <AppFuture at 0x1524c6e6de50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 52 has memoization hash b487121cdc717e50ed086360f2ef3ff6
parsl.dataflow.memoization INFO: Task 52 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4a00fc89-2d0c-4934-8233-484d2e4e47a6_isrForCrosstalkSources_898281_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4a00fc89-2d0c-4934-8233-484d2e4e47a6_isrForCrosstalkSources_898281_31.stderr
parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: <AppFuture at 0x1524c6c94290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 53 has memoization hash ca7d8015cf2e51224a9b0e6d11c1e6df
parsl.dataflow.memoization INFO: Task 53 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3ebae64f-a5bc-4441-a7f8-e2dd028bff49_isrForCrosstalkSources_898387_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3ebae64f-a5bc-4441-a7f8-e2dd028bff49_isrForCrosstalkSources_898387_42.stderr
parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: <AppFuture at 0x1524c6e66090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 0c92b1e815588d425db1b6338cb749fe
parsl.dataflow.memoization INFO: Task 54 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/bfd1bf58-2a19-473e-9555-068a03263f4a_isrForCrosstalkSources_898387_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/bfd1bf58-2a19-473e-9555-068a03263f4a_isrForCrosstalkSources_898387_14.stderr
parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: <AppFuture at 0x1524c6e67f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 90a31db2e5cfef3240148e93f11bba03
parsl.dataflow.memoization INFO: Task 55 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/01be10c7-f039-4fb5-b263-ac9df914ebc0_isrForCrosstalkSources_898282_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/01be10c7-f039-4fb5-b263-ac9df914ebc0_isrForCrosstalkSources_898282_60.stderr
parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: <AppFuture at 0x1524c6e5e190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 07893403f8feb2c5bbc249db7de7c19d
parsl.dataflow.memoization INFO: Task 56 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c97dbb63-c753-4d6d-971d-f0f69b1bbddc_isrForCrosstalkSources_898280_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c97dbb63-c753-4d6d-971d-f0f69b1bbddc_isrForCrosstalkSources_898280_14.stderr
parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: <AppFuture at 0x1524c6e5de50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 57 has memoization hash c42832a753ac8eb60baf04cb8a47c9c3
parsl.dataflow.memoization INFO: Task 57 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3e6f47e5-6772-41cb-80d1-851f43edfb14_isrForCrosstalkSources_898385_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3e6f47e5-6772-41cb-80d1-851f43edfb14_isrForCrosstalkSources_898385_9.stderr
parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: <AppFuture at 0x1524c6e5c650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 52679cf0b2dd089a100c18882d8912cc
parsl.dataflow.memoization INFO: Task 58 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ff49fdfd-3173-4eea-8c58-e6b3e3e461eb_isrForCrosstalkSources_898387_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ff49fdfd-3173-4eea-8c58-e6b3e3e461eb_isrForCrosstalkSources_898387_46.stderr
parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: <AppFuture at 0x1524c6d81190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 547db9253d8884640238321a4deb0737
parsl.dataflow.memoization INFO: Task 59 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a0b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/514e4fdc-5a5f-488b-8dbb-bfff6a9db519_isrForCrosstalkSources_898281_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/514e4fdc-5a5f-488b-8dbb-bfff6a9db519_isrForCrosstalkSources_898281_49.stderr
parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: <AppFuture at 0x1524c6cd96d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 1d39f9430738af9e04bb863f868196cb
parsl.dataflow.memoization INFO: Task 60 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a480e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/a0e225e9-81a4-491c-be26-1832651b0081_isrForCrosstalkSources_898282_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/a0e225e9-81a4-491c-be26-1832651b0081_isrForCrosstalkSources_898282_10.stderr
parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: <AppFuture at 0x1524c6e42f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 77b6f24758d7afbc678e52113a6d8f5e
parsl.dataflow.memoization INFO: Task 61 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/f8c033b4-f0f3-45ba-8c86-99d72ca31380_isrForCrosstalkSources_898387_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/f8c033b4-f0f3-45ba-8c86-99d72ca31380_isrForCrosstalkSources_898387_54.stderr
parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: <AppFuture at 0x1524c6e42cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 04634525748a86d4bb659d7d639a1954
parsl.dataflow.memoization INFO: Task 62 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/24c51d62-064e-49ab-b50c-e2563f91942e_isrForCrosstalkSources_898387_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/24c51d62-064e-49ab-b50c-e2563f91942e_isrForCrosstalkSources_898387_26.stderr
parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: <AppFuture at 0x1524c6e3d790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 3f58971163f5395fdbf6ae2dbac0f7ce
parsl.dataflow.memoization INFO: Task 63 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/8be8d598-abbb-463d-8dff-6c7fefda6790_isrForCrosstalkSources_898386_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/8be8d598-abbb-463d-8dff-6c7fefda6790_isrForCrosstalkSources_898386_60.stderr
parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: <AppFuture at 0x1524c6e3ddd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 15f0b396ebc3628671470927ed18ff80
parsl.dataflow.memoization INFO: Task 64 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a482c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/77ba637f-fef6-4452-87fb-26e18549ceef_isrForCrosstalkSources_898280_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/77ba637f-fef6-4452-87fb-26e18549ceef_isrForCrosstalkSources_898280_20.stderr
parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: <AppFuture at 0x1524c6e3e050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 0414502efd09092a6898f86e45eb4d69
parsl.dataflow.memoization INFO: Task 65 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/a2c4b4bd-abcf-47c4-847a-dccbd275fb58_isrForCrosstalkSources_898280_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/a2c4b4bd-abcf-47c4-847a-dccbd275fb58_isrForCrosstalkSources_898280_48.stderr
parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: <AppFuture at 0x1524c6e32710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 5c99b9d6f5825761198bac21a482044b
parsl.dataflow.memoization INFO: Task 66 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/b222d146-0c59-4c48-a286-ae23f4b95738_isrForCrosstalkSources_898386_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/b222d146-0c59-4c48-a286-ae23f4b95738_isrForCrosstalkSources_898386_57.stderr
parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: <AppFuture at 0x1524c6e33e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 67 has memoization hash b89e66aa51523b4a6faddf5322505880
parsl.dataflow.memoization INFO: Task 67 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1d4c7c27-928d-453d-953e-cf6f90f00fb8_isrForCrosstalkSources_898282_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1d4c7c27-928d-453d-953e-cf6f90f00fb8_isrForCrosstalkSources_898282_31.stderr
parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: <AppFuture at 0x1524c6e313d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 2ebd7dc9b28f41968eb8e9e3b6ee73e6
parsl.dataflow.memoization INFO: Task 68 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7baffd0c-6318-4a86-bf4c-03f939394d66_isrForCrosstalkSources_898281_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7baffd0c-6318-4a86-bf4c-03f939394d66_isrForCrosstalkSources_898281_9.stderr
parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: <AppFuture at 0x1524c6e27750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 69 has memoization hash c8051e81374da6f4adcbdb25a9bbab40
parsl.dataflow.memoization INFO: Task 69 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a485e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3605e328-a18a-4c69-acf4-38b9f86ef674_isrForCrosstalkSources_898387_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3605e328-a18a-4c69-acf4-38b9f86ef674_isrForCrosstalkSources_898387_33.stderr
parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: <AppFuture at 0x1524c6e24590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 029b95783a9863ec95eefdee130ec605
parsl.dataflow.memoization INFO: Task 70 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/035d1fb3-de5b-410d-a039-e3d6577a6965_isrForCrosstalkSources_898386_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/035d1fb3-de5b-410d-a039-e3d6577a6965_isrForCrosstalkSources_898386_1.stderr
parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: <AppFuture at 0x1524c6e24d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 71 has memoization hash b8c2fc3dc6175cb2a80712633cc93fc0
parsl.dataflow.memoization INFO: Task 71 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a487c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7cd41d53-6dd0-4b2e-adee-8e0a05c2e843_isrForCrosstalkSources_898385_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7cd41d53-6dd0-4b2e-adee-8e0a05c2e843_isrForCrosstalkSources_898385_29.stderr
parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: <AppFuture at 0x1524c6e218d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 23ebac5b4012236bffbd75e5f7bd8686
parsl.dataflow.memoization INFO: Task 72 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/d276d934-433b-4b0b-9a67-66158cb2e14a_isrForCrosstalkSources_898385_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/d276d934-433b-4b0b-9a67-66158cb2e14a_isrForCrosstalkSources_898385_10.stderr
parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: <AppFuture at 0x1524c6e23110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 88ea11a911edf92ea645580cafcacf5e
parsl.dataflow.memoization INFO: Task 73 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a489a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1d034d2f-e6da-492c-bf1d-6faf3522e076_isrForCrosstalkSources_898282_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1d034d2f-e6da-492c-bf1d-6faf3522e076_isrForCrosstalkSources_898282_27.stderr
parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: <AppFuture at 0x1524c6e22150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 6517f16bba02343fb6698177e06df193
parsl.dataflow.memoization INFO: Task 74 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/bd1b8ae4-f06f-469b-a43d-b9b75603a4db_isrForCrosstalkSources_898387_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/bd1b8ae4-f06f-469b-a43d-b9b75603a4db_isrForCrosstalkSources_898387_13.stderr
parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: <AppFuture at 0x1524c6e0da10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 7c6afe8d8143783e0086ef21ab09cdfa
parsl.dataflow.memoization INFO: Task 75 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d86c89e4-b375-49a2-b838-526aa51de49e_isrForCrosstalkSources_898282_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d86c89e4-b375-49a2-b838-526aa51de49e_isrForCrosstalkSources_898282_42.stderr
parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: <AppFuture at 0x1524c6e0f450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 76 has memoization hash a23a36d398ba4fd4b92bf2a304d0dbe8
parsl.dataflow.memoization INFO: Task 76 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e0554f2e-3eff-4502-89df-18eb0935c9a1_isrForCrosstalkSources_898281_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e0554f2e-3eff-4502-89df-18eb0935c9a1_isrForCrosstalkSources_898281_28.stderr
parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: <AppFuture at 0x1524c6e0f5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 158e677845e19b5baf5ed6f02fd8a5ec
parsl.dataflow.memoization INFO: Task 77 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7a3a47e3-12b3-4ded-9000-20c0712f6263_isrForCrosstalkSources_898281_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7a3a47e3-12b3-4ded-9000-20c0712f6263_isrForCrosstalkSources_898281_33.stderr
parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: <AppFuture at 0x1524c6e04b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 5117ee1456b4ba0329dd9246ce3e4978
parsl.dataflow.memoization INFO: Task 78 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/8e6c74a2-7be1-4e0d-a5d6-457a7c639163_isrForCrosstalkSources_898280_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/8e6c74a2-7be1-4e0d-a5d6-457a7c639163_isrForCrosstalkSources_898280_54.stderr
parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: <AppFuture at 0x1524c6e0d190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 1d1d79d1b5d60817053827a3f4908427
parsl.dataflow.memoization INFO: Task 79 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e0a76c0b-fe8f-4eac-a8a8-8c657903f457_isrForCrosstalkSources_898385_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e0a76c0b-fe8f-4eac-a8a8-8c657903f457_isrForCrosstalkSources_898385_34.stderr
parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: <AppFuture at 0x1524c6cccf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 3e81e065b1add20377bdc3c8ed660bf7
parsl.dataflow.memoization INFO: Task 80 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/fb675716-bc83-44df-83fb-7968e7ae12bb_isrForCrosstalkSources_898387_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/fb675716-bc83-44df-83fb-7968e7ae12bb_isrForCrosstalkSources_898387_3.stderr
parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: <AppFuture at 0x1524c6e07e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 40ef0f4666bee99f8aec0783b39a44b0
parsl.dataflow.memoization INFO: Task 81 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/0e902274-5344-458a-a19d-88be97803d9b_isrForCrosstalkSources_898280_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/0e902274-5344-458a-a19d-88be97803d9b_isrForCrosstalkSources_898280_57.stderr
parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: <AppFuture at 0x1524c6e40290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 82 has memoization hash f704ea47ddb38e4e424f02a7af9e0b7f
parsl.dataflow.memoization INFO: Task 82 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/f094ca45-3c8f-4336-8e85-9c9b9ad6c809_isrForCrosstalkSources_898280_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/f094ca45-3c8f-4336-8e85-9c9b9ad6c809_isrForCrosstalkSources_898280_55.stderr
parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: <AppFuture at 0x1524c6df8510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 6600472cc4538e0234e26df543213e9f
parsl.dataflow.memoization INFO: Task 83 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/388c99c9-6b57-42b4-a78d-be022dd70c4c_isrForCrosstalkSources_898281_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/388c99c9-6b57-42b4-a78d-be022dd70c4c_isrForCrosstalkSources_898281_30.stderr
parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: <AppFuture at 0x1524c6df9cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 84 has memoization hash deac855a19991494628ee1082bf86e24
parsl.dataflow.memoization INFO: Task 84 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/91e5e6c0-fb82-4690-8598-e503b7933d59_isrForCrosstalkSources_898385_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/91e5e6c0-fb82-4690-8598-e503b7933d59_isrForCrosstalkSources_898385_42.stderr
parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: <AppFuture at 0x1524c6dfbdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 85 has memoization hash c3041cb7991f40d668978555f41f3d58
parsl.dataflow.memoization INFO: Task 85 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/754d1e88-0451-43dc-a9d3-6f9d09545b00_isrForCrosstalkSources_898385_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/754d1e88-0451-43dc-a9d3-6f9d09545b00_isrForCrosstalkSources_898385_30.stderr
parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: <AppFuture at 0x1524c6debbd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 86 has memoization hash e1cc4ae0f2d57a1021ea6d88855a70f3
parsl.dataflow.memoization INFO: Task 86 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/2be23c92-824a-4045-aeee-61272cc2bfca_isrForCrosstalkSources_898280_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/2be23c92-824a-4045-aeee-61272cc2bfca_isrForCrosstalkSources_898280_40.stderr
parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: <AppFuture at 0x1524c6de9150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 79fa6cdaf0939371949eaeaa4b0126b9
parsl.dataflow.memoization INFO: Task 87 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a491c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3eef727a-0277-4e19-9752-824b77fc713f_isrForCrosstalkSources_898387_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3eef727a-0277-4e19-9752-824b77fc713f_isrForCrosstalkSources_898387_56.stderr
parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: <AppFuture at 0x1524c6de9c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 1b4d74d2f42036696fa99ea8d0613326
parsl.dataflow.memoization INFO: Task 88 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/2bbb5ec3-7871-4f36-ac2f-82bce59ab85c_isrForCrosstalkSources_898387_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/2bbb5ec3-7871-4f36-ac2f-82bce59ab85c_isrForCrosstalkSources_898387_53.stderr
parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: <AppFuture at 0x1524c6de35d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 1880fd1649c8e00817c90cde21562b0b
parsl.dataflow.memoization INFO: Task 89 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6b2163ec-620a-44f6-8052-b7173e55124f_isrForCrosstalkSources_898385_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6b2163ec-620a-44f6-8052-b7173e55124f_isrForCrosstalkSources_898385_8.stderr
parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: <AppFuture at 0x1524c6de1290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 1ec652b67163e34074342e802efaa100
parsl.dataflow.memoization INFO: Task 90 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a493a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/fb77a77c-5d96-44ed-9db3-6b030d582a77_isrForCrosstalkSources_898386_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/fb77a77c-5d96-44ed-9db3-6b030d582a77_isrForCrosstalkSources_898386_22.stderr
parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: <AppFuture at 0x1524c6de0050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 91 has memoization hash a150a765e43b7e0c6143becc98592490
parsl.dataflow.memoization INFO: Task 91 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/b82afdd6-28f7-41f2-a652-508bb35f3262_isrForCrosstalkSources_898385_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/b82afdd6-28f7-41f2-a652-508bb35f3262_isrForCrosstalkSources_898385_33.stderr
parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: <AppFuture at 0x1524c6de2d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 2ed14c1ef7e1f4a49220a2bdb17d580b
parsl.dataflow.memoization INFO: Task 92 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a494e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e651db0b-e51f-4e8a-a236-c0a75be2bd9a_isrForCrosstalkSources_898385_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e651db0b-e51f-4e8a-a236-c0a75be2bd9a_isrForCrosstalkSources_898385_4.stderr
parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: <AppFuture at 0x1524c6dd9010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 454fbc1a8d4c96f618e0e3b2e518eca7
parsl.dataflow.memoization INFO: Task 93 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/04e34fe9-99b1-4178-970d-47400b6cb66c_isrForCrosstalkSources_898385_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/04e34fe9-99b1-4178-970d-47400b6cb66c_isrForCrosstalkSources_898385_28.stderr
parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: <AppFuture at 0x1524c6dd96d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 992032f1a5296d9b21eb849afc11a406
parsl.dataflow.memoization INFO: Task 94 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/337bb5b3-b0a1-4186-b474-0db9cb2465f4_isrForCrosstalkSources_898280_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/337bb5b3-b0a1-4186-b474-0db9cb2465f4_isrForCrosstalkSources_898280_50.stderr
parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: <AppFuture at 0x1524c6dd8f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 139fbd3862fe536d711d389ff6812a03
parsl.dataflow.memoization INFO: Task 95 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a496c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0b9fd929-150a-49c4-bf69-4fa09e172c13_isrForCrosstalkSources_898386_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0b9fd929-150a-49c4-bf69-4fa09e172c13_isrForCrosstalkSources_898386_3.stderr
parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: <AppFuture at 0x1524c6dd99d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 46bfc325a8642502f1e9e1b5a1f7960f
parsl.dataflow.memoization INFO: Task 96 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/4a1fd99d-cc97-4243-ada9-ca367939300c_isrForCrosstalkSources_898387_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/4a1fd99d-cc97-4243-ada9-ca367939300c_isrForCrosstalkSources_898387_12.stderr
parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: <AppFuture at 0x1524c6dcfed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 178eca08806e2172676896c0cfc5e521
parsl.dataflow.memoization INFO: Task 97 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e935ec2d-3239-44c9-ac60-069079cf2bbb_isrForCrosstalkSources_898280_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e935ec2d-3239-44c9-ac60-069079cf2bbb_isrForCrosstalkSources_898280_43.stderr
parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: <AppFuture at 0x1524c6dcc310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 2dc072d97b344741198a7907d4523277
parsl.dataflow.memoization INFO: Task 98 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/4f3fc0b9-f8d4-4389-8d38-8057ff6eade3_isrForCrosstalkSources_898387_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/4f3fc0b9-f8d4-4389-8d38-8057ff6eade3_isrForCrosstalkSources_898387_8.stderr
parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: <AppFuture at 0x1524c6dcf990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 99 has memoization hash e578bd0571df2d50e020af5dc2bb9024
parsl.dataflow.memoization INFO: Task 99 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a498a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/38fecc07-a335-4cb0-a785-108cffe74cb7_isrForCrosstalkSources_898280_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/38fecc07-a335-4cb0-a785-108cffe74cb7_isrForCrosstalkSources_898280_53.stderr
parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: <AppFuture at 0x1524c6dcc610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 100 has memoization hash bd282fa8573fb0ea1b02d58c48b6d230
parsl.dataflow.memoization INFO: Task 100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a499e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/4e63639d-6651-4a4a-ac9f-bff48ec7cfe6_isrForCrosstalkSources_898386_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/4e63639d-6651-4a4a-ac9f-bff48ec7cfe6_isrForCrosstalkSources_898386_30.stderr
parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: <AppFuture at 0x1524c6dcc8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 81e528c91fb052577ac2fb42967c9b04
parsl.dataflow.memoization INFO: Task 101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/affd6abb-27f7-4a88-9f80-fc5c5cb58dea_isrForCrosstalkSources_898385_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/affd6abb-27f7-4a88-9f80-fc5c5cb58dea_isrForCrosstalkSources_898385_59.stderr
parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: <AppFuture at 0x1524c6dc9390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 3b08d859e803673aa97b433fa98dfb1e
parsl.dataflow.memoization INFO: Task 102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/edaa34e5-083d-4208-88d2-728fef6f79dd_isrForCrosstalkSources_898386_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/edaa34e5-083d-4208-88d2-728fef6f79dd_isrForCrosstalkSources_898386_11.stderr
parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: <AppFuture at 0x1524c6dcbb50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 103 has memoization hash fba0c96e304921360edd7785e3c19831
parsl.dataflow.memoization INFO: Task 103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7e1e93f5-2130-405b-869a-14797fb47763_isrForCrosstalkSources_898281_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7e1e93f5-2130-405b-869a-14797fb47763_isrForCrosstalkSources_898281_14.stderr
parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: <AppFuture at 0x1524c6dcbf90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 11b3bf88cf2994f0c94bc6ab12fa53ed
parsl.dataflow.memoization INFO: Task 104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/13503b45-f338-49bb-b518-c1883eeaf47f_isrForCrosstalkSources_898387_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/13503b45-f338-49bb-b518-c1883eeaf47f_isrForCrosstalkSources_898387_10.stderr
parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: <AppFuture at 0x1524c7058350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 105 has memoization hash addef5db32c507019249e235e287fa07
parsl.dataflow.memoization INFO: Task 105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/d2f86bba-e154-4264-88f9-9c55c44bd70e_isrForCrosstalkSources_898386_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/d2f86bba-e154-4264-88f9-9c55c44bd70e_isrForCrosstalkSources_898386_10.stderr
parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: <AppFuture at 0x1524c6dc9810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 57a114670ed709fe7b2f0fae934b25d4
parsl.dataflow.memoization INFO: Task 106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/87d1e268-4ace-4936-bd9c-0b0c2968c098_isrForCrosstalkSources_898385_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/87d1e268-4ace-4936-bd9c-0b0c2968c098_isrForCrosstalkSources_898385_11.stderr
parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: <AppFuture at 0x1524c6dc1e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 107 has memoization hash fd3236decffe114f058d1c9a83420f1a
parsl.dataflow.memoization INFO: Task 107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5d669546-ba4d-463f-b32a-bdcf48b34860_isrForCrosstalkSources_898385_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5d669546-ba4d-463f-b32a-bdcf48b34860_isrForCrosstalkSources_898385_13.stderr
parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: <AppFuture at 0x1524c6dc16d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 88e6e1d4e87f81d5957daf4e73cbf0f1
parsl.dataflow.memoization INFO: Task 108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/56b7d0a8-d906-4941-a6ff-fb366aeb9055_isrForCrosstalkSources_898387_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/56b7d0a8-d906-4941-a6ff-fb366aeb9055_isrForCrosstalkSources_898387_2.stderr
parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: <AppFuture at 0x1524c6dc3c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 109 has memoization hash ca75def372d49171b50e8a9af9872a73
parsl.dataflow.memoization INFO: Task 109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a49ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/9825ae25-8802-4976-b6c6-d8c357848e9f_isrForCrosstalkSources_898386_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/9825ae25-8802-4976-b6c6-d8c357848e9f_isrForCrosstalkSources_898386_18.stderr
parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: <AppFuture at 0x1524c6dc0a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 066cb37aa10f3829aa68c394153e9f22
parsl.dataflow.memoization INFO: Task 110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/559cdd8b-168b-424f-a57c-733f119f8fea_isrForCrosstalkSources_898280_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/559cdd8b-168b-424f-a57c-733f119f8fea_isrForCrosstalkSources_898280_21.stderr
parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: <AppFuture at 0x1524c6dc2e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 14df738597db02a2b7a1569ce332103d
parsl.dataflow.memoization INFO: Task 111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/38ffc2a8-afe2-4e6e-b032-d673eeb4c6fc_isrForCrosstalkSources_898280_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/38ffc2a8-afe2-4e6e-b032-d673eeb4c6fc_isrForCrosstalkSources_898280_31.stderr
parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: <AppFuture at 0x1524c6db8490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 93e2289944be28ba6af2562e12f64c93
parsl.dataflow.memoization INFO: Task 112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/74ab14bd-3354-4b28-b8bb-0b8ab86a2923_isrForCrosstalkSources_898281_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/74ab14bd-3354-4b28-b8bb-0b8ab86a2923_isrForCrosstalkSources_898281_20.stderr
parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: <AppFuture at 0x1524c6dbb850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 2b2d57b9f55a3fcb7075ebfd28043728
parsl.dataflow.memoization INFO: Task 113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/46db20f2-d2d2-453e-9e87-92a064193082_isrForCrosstalkSources_898386_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/46db20f2-d2d2-453e-9e87-92a064193082_isrForCrosstalkSources_898386_31.stderr
parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: <AppFuture at 0x1524c6db8410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 84cb046ce91f5aea46ff26029f36f980
parsl.dataflow.memoization INFO: Task 114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/1d681fb1-7a5a-4a1a-96a0-d767dc6b29b8_isrForCrosstalkSources_898280_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/1d681fb1-7a5a-4a1a-96a0-d767dc6b29b8_isrForCrosstalkSources_898280_6.stderr
parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: <AppFuture at 0x1524c6db90d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 115 has memoization hash cb22293e46ecfda6a8a67ffa95c34b84
parsl.dataflow.memoization INFO: Task 115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/27a1bdb7-b5de-41b8-94d8-005606afd4c8_isrForCrosstalkSources_898280_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/27a1bdb7-b5de-41b8-94d8-005606afd4c8_isrForCrosstalkSources_898280_35.stderr
parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: <AppFuture at 0x1524c6db8cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 3592fb4dc7a65d16ae68ba958ae11985
parsl.dataflow.memoization INFO: Task 116 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5ef28667-acd8-4a08-8349-03f895b2823b_isrForCrosstalkSources_898385_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5ef28667-acd8-4a08-8349-03f895b2823b_isrForCrosstalkSources_898385_49.stderr
parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: <AppFuture at 0x1524c6db7950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 6d8656a10d7d2ec8571f150778153bb1
parsl.dataflow.memoization INFO: Task 117 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/37cbd2c8-d0f8-4f8f-a974-cb7cd65c6612_isrForCrosstalkSources_898386_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/37cbd2c8-d0f8-4f8f-a974-cb7cd65c6612_isrForCrosstalkSources_898386_17.stderr
parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: <AppFuture at 0x1524c6db5bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 118 has memoization hash ff31b6f38e5b67e41804295b3992b3ae
parsl.dataflow.memoization INFO: Task 118 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2c54af98-b1ef-4c6d-9c81-c3ad88290728_isrForCrosstalkSources_898281_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2c54af98-b1ef-4c6d-9c81-c3ad88290728_isrForCrosstalkSources_898281_62.stderr
parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: <AppFuture at 0x1524c6db5710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 119 has memoization hash d89694b73d74bbfa6e5fd33878ce8be4
parsl.dataflow.memoization INFO: Task 119 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/db10c3e5-56bf-4a22-a702-da47f9f1f954_isrForCrosstalkSources_898281_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/db10c3e5-56bf-4a22-a702-da47f9f1f954_isrForCrosstalkSources_898281_7.stderr
parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: <AppFuture at 0x1524c6db7610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 4fa45a9f370d478a422c494509715c9a
parsl.dataflow.memoization INFO: Task 120 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/1b3e33d8-7946-4842-afc3-08f90357d88f_isrForCrosstalkSources_898280_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/1b3e33d8-7946-4842-afc3-08f90357d88f_isrForCrosstalkSources_898280_52.stderr
parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: <AppFuture at 0x1524c6d97790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 121 has memoization hash b029a8d938c78a08097e53aca9fe6b9a
parsl.dataflow.memoization INFO: Task 121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/b3c0a56d-7b3c-4874-8890-6c8cba4dbee0_isrForCrosstalkSources_898385_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/b3c0a56d-7b3c-4874-8890-6c8cba4dbee0_isrForCrosstalkSources_898385_22.stderr
parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: <AppFuture at 0x1524c6dc3610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 746dc3a759211ed54e2cd42e5a019520
parsl.dataflow.memoization INFO: Task 122 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b435c2ac-4290-4bf5-8e76-0a45f881eadb_isrForCrosstalkSources_898282_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b435c2ac-4290-4bf5-8e76-0a45f881eadb_isrForCrosstalkSources_898282_16.stderr
parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: <AppFuture at 0x1524d124a950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 37b6020dc871ea6e580e7a25b51f20a5
parsl.dataflow.memoization INFO: Task 123 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5d2bbb3c-c7d1-47e7-886e-50087a7e336a_isrForCrosstalkSources_898385_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5d2bbb3c-c7d1-47e7-886e-50087a7e336a_isrForCrosstalkSources_898385_61.stderr
parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: <AppFuture at 0x1524d124a7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 124 has memoization hash f015d75f50eba1dcce05eb46fca89c54
parsl.dataflow.memoization INFO: Task 124 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/dc82db18-28d2-4482-a06c-f9380187a673_isrForCrosstalkSources_898386_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/dc82db18-28d2-4482-a06c-f9380187a673_isrForCrosstalkSources_898386_45.stderr
parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: <AppFuture at 0x1524d1249c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 00896763f662c3926b479cc9b4e5a858
parsl.dataflow.memoization INFO: Task 125 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b05d69a7-b7fb-4d30-a88d-970c0af1d11d_isrForCrosstalkSources_898282_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b05d69a7-b7fb-4d30-a88d-970c0af1d11d_isrForCrosstalkSources_898282_47.stderr
parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: <AppFuture at 0x1524d124a250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 6dd1a4cffff823606c864479921762c5
parsl.dataflow.memoization INFO: Task 126 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2d298252-6b82-4eb2-9f21-9696a1f4752f_isrForCrosstalkSources_898282_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2d298252-6b82-4eb2-9f21-9696a1f4752f_isrForCrosstalkSources_898282_58.stderr
parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: <AppFuture at 0x1524d1248c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 12f2cf69c0dce8e3d83b0c6da05a0c91
parsl.dataflow.memoization INFO: Task 127 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/48ccb5a2-1065-4ea7-bd1e-dc18dda8cd83_isrForCrosstalkSources_898387_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/48ccb5a2-1065-4ea7-bd1e-dc18dda8cd83_isrForCrosstalkSources_898387_48.stderr
parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: <AppFuture at 0x1524d1248650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 128 has memoization hash c962aba4980ee47a89e9076560408a7c
parsl.dataflow.memoization INFO: Task 128 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/109496d3-91f7-4400-a0ae-37fef025e2cc_isrForCrosstalkSources_898280_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/109496d3-91f7-4400-a0ae-37fef025e2cc_isrForCrosstalkSources_898280_5.stderr
parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: <AppFuture at 0x1524d1248490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 129 has memoization hash beef2f4b92772d63164372699efc61ac
parsl.dataflow.memoization INFO: Task 129 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/a2910115-4816-49fc-99be-328dbd5cbacb_isrForCrosstalkSources_898280_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/a2910115-4816-49fc-99be-328dbd5cbacb_isrForCrosstalkSources_898280_59.stderr
parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: <AppFuture at 0x1524d12fc610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 508f948e3212ec9e3c2f3b4dd2602bae
parsl.dataflow.memoization INFO: Task 130 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f5a33d6b-32fe-410f-8d84-584a184e9220_isrForCrosstalkSources_898386_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f5a33d6b-32fe-410f-8d84-584a184e9220_isrForCrosstalkSources_898386_33.stderr
parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: <AppFuture at 0x1524d12fffd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 14ea10625b0a9408bc77f4b0f56fdeeb
parsl.dataflow.memoization INFO: Task 131 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/7fe59330-e716-44db-8d07-4193d49cebe1_isrForCrosstalkSources_898280_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/7fe59330-e716-44db-8d07-4193d49cebe1_isrForCrosstalkSources_898280_56.stderr
parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: <AppFuture at 0x1524d124abd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 804d810e542c513696c11d619b045c37
parsl.dataflow.memoization INFO: Task 132 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f5544d53-eeac-4d1a-be75-f7243c0c91d0_isrForCrosstalkSources_898386_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f5544d53-eeac-4d1a-be75-f7243c0c91d0_isrForCrosstalkSources_898386_14.stderr
parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: <AppFuture at 0x1524c6da8350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 133 has memoization hash e5ffa39e85260f2e3dfef2f881f22d20
parsl.dataflow.memoization INFO: Task 133 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/56467881-8553-4be9-9662-fcef57f6cb85_isrForCrosstalkSources_898387_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/56467881-8553-4be9-9662-fcef57f6cb85_isrForCrosstalkSources_898387_50.stderr
parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: <AppFuture at 0x1524c6da85d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 2bbe4913834eee1dd1701ea686ee4424
parsl.dataflow.memoization INFO: Task 134 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/45b6ec5b-bdff-4572-9f95-b1f8e39f3746_isrForCrosstalkSources_898387_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/45b6ec5b-bdff-4572-9f95-b1f8e39f3746_isrForCrosstalkSources_898387_32.stderr
parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: <AppFuture at 0x1524c6dab510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 135 has memoization hash a64aac11bac6d44d1520d7b53b759bf1
parsl.dataflow.memoization INFO: Task 135 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/91be2ad1-db28-4c64-924f-f151b582d7a9_isrForCrosstalkSources_898385_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/91be2ad1-db28-4c64-924f-f151b582d7a9_isrForCrosstalkSources_898385_50.stderr
parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: <AppFuture at 0x1524c6b39410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 136 has memoization hash f0cfd825556a087234ebb1d207224bd7
parsl.dataflow.memoization INFO: Task 136 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/95daf612-8458-4f07-adfd-15a9cea0da9a_isrForCrosstalkSources_898281_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/95daf612-8458-4f07-adfd-15a9cea0da9a_isrForCrosstalkSources_898281_51.stderr
parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: <AppFuture at 0x1524c6b3ae90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 2e975c1eee28d3db3644152d246a4d5b
parsl.dataflow.memoization INFO: Task 137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ab9d14cc-0c27-4c91-a4da-0135758cfe47_isrForCrosstalkSources_898387_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ab9d14cc-0c27-4c91-a4da-0135758cfe47_isrForCrosstalkSources_898387_60.stderr
parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: <AppFuture at 0x1524c6b3b690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 21f1e089ea742186b8a4f609394b7e85
parsl.dataflow.memoization INFO: Task 138 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/623f79be-83be-412d-9028-9239d10a717f_isrForCrosstalkSources_898386_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/623f79be-83be-412d-9028-9239d10a717f_isrForCrosstalkSources_898386_48.stderr
parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: <AppFuture at 0x1524c6b38790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 139 has memoization hash b961b6c2d534c79a235903f941fc7f39
parsl.dataflow.memoization INFO: Task 139 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7c4fa937-6298-4033-943e-26a0d765e94c_isrForCrosstalkSources_898387_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7c4fa937-6298-4033-943e-26a0d765e94c_isrForCrosstalkSources_898387_40.stderr
parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: <AppFuture at 0x1524c6ca7b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 72c0b0a97ee429e74d34d574e3316063
parsl.dataflow.memoization INFO: Task 140 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/aaab69eb-7387-4823-b94d-457e92b6f9c3_isrForCrosstalkSources_898280_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/aaab69eb-7387-4823-b94d-457e92b6f9c3_isrForCrosstalkSources_898280_1.stderr
parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: <AppFuture at 0x1524c69dfcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 4ce6ea97e0b4a225e962f251a44917de
parsl.dataflow.memoization INFO: Task 141 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7363ec24-9a59-4e77-bc84-0a0257ed0f35_isrForCrosstalkSources_898385_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7363ec24-9a59-4e77-bc84-0a0257ed0f35_isrForCrosstalkSources_898385_14.stderr
parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: <AppFuture at 0x1524c69dcf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 0a6f604fb4f3430acf5a2e367b7600c6
parsl.dataflow.memoization INFO: Task 142 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e1180f62-bce8-4c0e-b97e-c4a114357a64_isrForCrosstalkSources_898280_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e1180f62-bce8-4c0e-b97e-c4a114357a64_isrForCrosstalkSources_898280_24.stderr
parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: <AppFuture at 0x1524c69dc150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 08276e167965e02656d4387041ec7aff
parsl.dataflow.memoization INFO: Task 143 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2df9bfa7-36e5-430e-80e2-f7745c7aa294_isrForCrosstalkSources_898281_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2df9bfa7-36e5-430e-80e2-f7745c7aa294_isrForCrosstalkSources_898281_15.stderr
parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: <AppFuture at 0x1524c69ec650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 144 has memoization hash aaeb3802b3a44e256c6626505503b3cf
parsl.dataflow.memoization INFO: Task 144 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/f0bb9889-4558-4728-b09b-562ef77f6994_isrForCrosstalkSources_898280_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/f0bb9889-4558-4728-b09b-562ef77f6994_isrForCrosstalkSources_898280_23.stderr
parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: <AppFuture at 0x1524c69ec2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 727cd2f56b3ce3f56f8920aa19f54c54
parsl.dataflow.memoization INFO: Task 145 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2ebe8bdf-f7fa-4a24-9664-4cf278e591eb_isrForCrosstalkSources_898281_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2ebe8bdf-f7fa-4a24-9664-4cf278e591eb_isrForCrosstalkSources_898281_29.stderr
parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: <AppFuture at 0x1524c69ed3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 146 has memoization hash a63890c7b5d4ecad48993e12ec6040b6
parsl.dataflow.memoization INFO: Task 146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/28347066-a84e-49ee-bb2b-72198923c3b2_isrForCrosstalkSources_898280_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/28347066-a84e-49ee-bb2b-72198923c3b2_isrForCrosstalkSources_898280_7.stderr
parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: <AppFuture at 0x1524c69ede90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 1dbfe3a254967cee5d142192d169ff1f
parsl.dataflow.memoization INFO: Task 147 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2cc4d09b-3f08-4f49-941d-4a71450eb8dd_isrForCrosstalkSources_898282_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2cc4d09b-3f08-4f49-941d-4a71450eb8dd_isrForCrosstalkSources_898282_12.stderr
parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: <AppFuture at 0x1524c69eee90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 92b92d57dfe9688ed3b1f5b686fae6d4
parsl.dataflow.memoization INFO: Task 148 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/31206e2b-a2b7-4114-ac85-1f6d903a7284_isrForCrosstalkSources_898282_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/31206e2b-a2b7-4114-ac85-1f6d903a7284_isrForCrosstalkSources_898282_37.stderr
parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: <AppFuture at 0x1524c69efed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 149 has memoization hash c01827d37e5f65fffb4dff2a3b8b3519
parsl.dataflow.memoization INFO: Task 149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c300cccb-912f-4964-a2cc-b01116971fe9_isrForCrosstalkSources_898282_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c300cccb-912f-4964-a2cc-b01116971fe9_isrForCrosstalkSources_898282_3.stderr
parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: <AppFuture at 0x1524c546c4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 55cd500f18fdc62c3b741d727da31637
parsl.dataflow.memoization INFO: Task 150 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/9f05c5c3-decb-401c-884f-0d55c22afa35_isrForCrosstalkSources_898281_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/9f05c5c3-decb-401c-884f-0d55c22afa35_isrForCrosstalkSources_898281_56.stderr
parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: <AppFuture at 0x1524c546d510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 5a630e700fa51208409ea3d1f71ddfc0
parsl.dataflow.memoization INFO: Task 151 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/dcc4f17a-a929-4b77-8cf1-3d0a7198c754_isrForCrosstalkSources_898385_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/dcc4f17a-a929-4b77-8cf1-3d0a7198c754_isrForCrosstalkSources_898385_35.stderr
parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: <AppFuture at 0x1524c546d850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 152 has memoization hash fba5a1028a1a629d1ca36ae63f0c1706
parsl.dataflow.memoization INFO: Task 152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/eaf86225-fdba-43b2-8b11-2cead9adf579_isrForCrosstalkSources_898387_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/eaf86225-fdba-43b2-8b11-2cead9adf579_isrForCrosstalkSources_898387_30.stderr
parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: <AppFuture at 0x1524c546df50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 77f5718780bec33120caabd46514c62b
parsl.dataflow.memoization INFO: Task 153 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/893ac501-c1fb-4933-b2cb-f4a5ee430188_isrForCrosstalkSources_898281_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/893ac501-c1fb-4933-b2cb-f4a5ee430188_isrForCrosstalkSources_898281_26.stderr
parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: <AppFuture at 0x1524c546e910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 0315fcd1a0ad9e854715a85d0b0eb8cf
parsl.dataflow.memoization INFO: Task 154 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2223fc0b-c9cc-4285-a76e-1d94426a771a_isrForCrosstalkSources_898281_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2223fc0b-c9cc-4285-a76e-1d94426a771a_isrForCrosstalkSources_898281_34.stderr
parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: <AppFuture at 0x1524c546f410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 155 has memoization hash c8a9aca783f11d8993efcb904e79865d
parsl.dataflow.memoization INFO: Task 155 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/149b2272-cb8c-4cf1-b0d7-dd93deef8bc8_isrForCrosstalkSources_898282_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/149b2272-cb8c-4cf1-b0d7-dd93deef8bc8_isrForCrosstalkSources_898282_50.stderr
parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: <AppFuture at 0x1524c546f6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 156 has memoization hash eedcc0169bcca30c9666bffeb7c231e9
parsl.dataflow.memoization INFO: Task 156 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/768f2d09-f2d6-4598-b968-1add7587a67c_isrForCrosstalkSources_898281_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/768f2d09-f2d6-4598-b968-1add7587a67c_isrForCrosstalkSources_898281_23.stderr
parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: <AppFuture at 0x1524c546ff50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 562516bf78a8177b03d2fee73a0a478e
parsl.dataflow.memoization INFO: Task 157 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/1c4641fa-d0ca-4396-8e53-9f30e0fbdeac_isrForCrosstalkSources_898386_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/1c4641fa-d0ca-4396-8e53-9f30e0fbdeac_isrForCrosstalkSources_898386_62.stderr
parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: <AppFuture at 0x1524c54790d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 596b71009cba0cefc386a60bf17ee1f7
parsl.dataflow.memoization INFO: Task 158 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/289f3346-fc10-457d-8055-6ab9ed26ef78_isrForCrosstalkSources_898387_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/289f3346-fc10-457d-8055-6ab9ed26ef78_isrForCrosstalkSources_898387_51.stderr
parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: <AppFuture at 0x1524c5478910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 159 has memoization hash db838e504436555fe951748aefe3eaec
parsl.dataflow.memoization INFO: Task 159 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/8fa3d833-3e35-4eb1-8cf8-099152633dc0_isrForCrosstalkSources_898281_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/8fa3d833-3e35-4eb1-8cf8-099152633dc0_isrForCrosstalkSources_898281_41.stderr
parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: <AppFuture at 0x1524c5479f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 9ba5f59b630b6ae56f852a8d1b302d48
parsl.dataflow.memoization INFO: Task 160 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a4bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/86c28f90-d604-4217-8aa0-f14dfcb2fa94_isrForCrosstalkSources_898385_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/86c28f90-d604-4217-8aa0-f14dfcb2fa94_isrForCrosstalkSources_898385_48.stderr
parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: <AppFuture at 0x1524c547a210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 161 has memoization hash c4284008f192758e49ff44c98d1d4f5b
parsl.dataflow.memoization INFO: Task 161 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c6a48040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ba2e802c-a399-4fe5-a11d-b88bab4a841d_isrForCrosstalkSources_898280_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ba2e802c-a399-4fe5-a11d-b88bab4a841d_isrForCrosstalkSources_898280_62.stderr
parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: <AppFuture at 0x1524c547ac90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 162 has memoization hash a07125a4c7426d650555891f3b89c36f
parsl.dataflow.memoization INFO: Task 162 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54840e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/9cf61144-d391-4ea5-99a5-d0ca96505e26_isrForCrosstalkSources_898281_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/9cf61144-d391-4ea5-99a5-d0ca96505e26_isrForCrosstalkSources_898281_50.stderr
parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: <AppFuture at 0x1524c547bdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 0fa242fcf0027821dd6db22c6a742695
parsl.dataflow.memoization INFO: Task 163 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/5ea83cd1-52df-411d-9df3-707c0895a288_isrForCrosstalkSources_898387_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/5ea83cd1-52df-411d-9df3-707c0895a288_isrForCrosstalkSources_898387_49.stderr
parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: <AppFuture at 0x1524c547bc50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 164 has memoization hash dd02994b25152380a15345f6351bb778
parsl.dataflow.memoization INFO: Task 164 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ab2c18e9-8b74-4a1c-99cd-60a28338ca7d_isrForCrosstalkSources_898387_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ab2c18e9-8b74-4a1c-99cd-60a28338ca7d_isrForCrosstalkSources_898387_36.stderr
parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: <AppFuture at 0x1524c5488c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 5664e94fd38bc5d0cc5118b08a7897c9
parsl.dataflow.memoization INFO: Task 165 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54842c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2e548571-7f9e-4de3-8dc8-9eb7139b646a_isrForCrosstalkSources_898386_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2e548571-7f9e-4de3-8dc8-9eb7139b646a_isrForCrosstalkSources_898386_16.stderr
parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: <AppFuture at 0x1524c5489550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 166 has memoization hash f984f422cf9ef9932339f492949c1ce5
parsl.dataflow.memoization INFO: Task 166 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/c02b9ba2-40ee-4acc-854f-88f517d25d02_isrForCrosstalkSources_898281_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/c02b9ba2-40ee-4acc-854f-88f517d25d02_isrForCrosstalkSources_898281_10.stderr
parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: <AppFuture at 0x1524c5489090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 167 has memoization hash bef8afe43f044a5291ebf9cfe94ac4f8
parsl.dataflow.memoization INFO: Task 167 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/0874b0f2-423a-4da0-9a9f-334cf0e75697_isrForCrosstalkSources_898387_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/0874b0f2-423a-4da0-9a9f-334cf0e75697_isrForCrosstalkSources_898387_57.stderr
parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: <AppFuture at 0x1524c548a410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 398b9bbdbb866fee980f2abf20867b36
parsl.dataflow.memoization INFO: Task 168 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6774ce4f-ef02-4960-a55c-314fb976987f_isrForCrosstalkSources_898385_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6774ce4f-ef02-4960-a55c-314fb976987f_isrForCrosstalkSources_898385_53.stderr
parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: <AppFuture at 0x1524c548ac50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 73a892624061891803c9d89fee918a7b
parsl.dataflow.memoization INFO: Task 169 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54844a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3fde7376-9e80-4012-9ee3-a39c2d5b4aff_isrForCrosstalkSources_898385_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3fde7376-9e80-4012-9ee3-a39c2d5b4aff_isrForCrosstalkSources_898385_52.stderr
parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: <AppFuture at 0x1524c548b790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 2384e0f88b2bbb8273d6b32ee874979c
parsl.dataflow.memoization INFO: Task 170 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54845e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e9ceb12c-396f-4d6f-9af3-625b7d4cb7f1_isrForCrosstalkSources_898385_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e9ceb12c-396f-4d6f-9af3-625b7d4cb7f1_isrForCrosstalkSources_898385_36.stderr
parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: <AppFuture at 0x1524c548b410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 0ac7373a8f14775b631db401cd043466
parsl.dataflow.memoization INFO: Task 171 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/0488bae8-4d42-40cd-acd1-9aac54d1853f_isrForCrosstalkSources_898387_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/0488bae8-4d42-40cd-acd1-9aac54d1853f_isrForCrosstalkSources_898387_35.stderr
parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: <AppFuture at 0x1524c6de0a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 7c54014058bf8f70f551e538a7e4a55d
parsl.dataflow.memoization INFO: Task 172 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5a416810-3e06-423c-95e9-4dfba26671c1_isrForCrosstalkSources_898385_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5a416810-3e06-423c-95e9-4dfba26671c1_isrForCrosstalkSources_898385_45.stderr
parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: <AppFuture at 0x1524c5488a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 0d3c43c007a6f23a7613667707583e79
parsl.dataflow.memoization INFO: Task 173 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/2f210879-95e0-4c42-8028-7681ad7ad1a7_isrForCrosstalkSources_898385_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/2f210879-95e0-4c42-8028-7681ad7ad1a7_isrForCrosstalkSources_898385_6.stderr
parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: <AppFuture at 0x1524c5499990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 235d1a310956e8b8e7da580cb66ed899
parsl.dataflow.memoization INFO: Task 174 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54847c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3767a720-eef6-426d-9b1d-61dcc76a004e_isrForCrosstalkSources_898387_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3767a720-eef6-426d-9b1d-61dcc76a004e_isrForCrosstalkSources_898387_59.stderr
parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: <AppFuture at 0x1524c549aad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 7dadcfe5be31fac85f7811bc5be67b30
parsl.dataflow.memoization INFO: Task 175 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/71c9d317-bdb8-4a2b-8939-822014a5b255_isrForCrosstalkSources_898386_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/71c9d317-bdb8-4a2b-8939-822014a5b255_isrForCrosstalkSources_898386_20.stderr
parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: <AppFuture at 0x1524c5499e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 55c231b14bcbe387928b70b04720daef
parsl.dataflow.memoization INFO: Task 176 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54849a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/3ffbb471-e6e7-4e01-b5e0-9bd7cb1e1bb1_isrForCrosstalkSources_898281_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/3ffbb471-e6e7-4e01-b5e0-9bd7cb1e1bb1_isrForCrosstalkSources_898281_11.stderr
parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: <AppFuture at 0x1524c549bcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 177 has memoization hash a6410e22947419d199bf454b8eb3568c
parsl.dataflow.memoization INFO: Task 177 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/816524f7-38b7-463a-94a4-bd50121459dc_isrForCrosstalkSources_898281_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/816524f7-38b7-463a-94a4-bd50121459dc_isrForCrosstalkSources_898281_12.stderr
parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: <AppFuture at 0x1524c549bc50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 52ad14e025dcab58efefbd1492a57dc3
parsl.dataflow.memoization INFO: Task 178 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f1266bdc-a44c-4dbe-a88a-23be5ab7759f_isrForCrosstalkSources_898386_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f1266bdc-a44c-4dbe-a88a-23be5ab7759f_isrForCrosstalkSources_898386_21.stderr
parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: <AppFuture at 0x1524c549b610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 179 has memoization hash a23cff98c736c2d6abd17f3469998d23
parsl.dataflow.memoization INFO: Task 179 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/7eac29c7-e6c0-4ee9-9bff-410c1ac65464_isrForCrosstalkSources_898386_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/7eac29c7-e6c0-4ee9-9bff-410c1ac65464_isrForCrosstalkSources_898386_23.stderr
parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: <AppFuture at 0x1524c5494b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 20165d74ee47cba94dbf0b9932a46e07
parsl.dataflow.memoization INFO: Task 180 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7dbff09f-527d-4bae-b6ef-a0e0cb62c63f_isrForCrosstalkSources_898385_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7dbff09f-527d-4bae-b6ef-a0e0cb62c63f_isrForCrosstalkSources_898385_39.stderr
parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: <AppFuture at 0x1524c5495210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 181 has memoization hash dc5d0457a7005fbe6e220dc389b337ad
parsl.dataflow.memoization INFO: Task 181 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/85f5afe2-7f69-4df8-b154-523d4378d3f5_isrForCrosstalkSources_898282_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/85f5afe2-7f69-4df8-b154-523d4378d3f5_isrForCrosstalkSources_898282_39.stderr
parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: <AppFuture at 0x1524c5495cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 182 has memoization hash edb1344cb881e45cbfa49bebc91b1c35
parsl.dataflow.memoization INFO: Task 182 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/4fc96d1d-deb2-47d0-a7cc-4ccf73538d43_isrForCrosstalkSources_898385_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/4fc96d1d-deb2-47d0-a7cc-4ccf73538d43_isrForCrosstalkSources_898385_51.stderr
parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: <AppFuture at 0x1524c54963d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 183 has memoization hash c4dc4f05ad1e929c14ead4fa9c578b6c
parsl.dataflow.memoization INFO: Task 183 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e175c2e5-b485-4443-866d-2c3c199ae18e_isrForCrosstalkSources_898281_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e175c2e5-b485-4443-866d-2c3c199ae18e_isrForCrosstalkSources_898281_46.stderr
parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: <AppFuture at 0x1524c5496990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 184 has memoization hash e8bf6e39b52b4efeacd856f4eba3cb56
parsl.dataflow.memoization INFO: Task 184 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/adcb97b7-fada-460c-90f6-949785ce6356_isrForCrosstalkSources_898280_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/adcb97b7-fada-460c-90f6-949785ce6356_isrForCrosstalkSources_898280_49.stderr
parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: <AppFuture at 0x1524c5496150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 8588646e2344b7424d71f00665a95c40
parsl.dataflow.memoization INFO: Task 185 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/bdf9e73b-fa9b-471a-984d-2267cf5b8ae1_isrForCrosstalkSources_898386_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/bdf9e73b-fa9b-471a-984d-2267cf5b8ae1_isrForCrosstalkSources_898386_19.stderr
parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: <AppFuture at 0x1524c5497710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 35efda2a7955b53e3d4c5d3982451552
parsl.dataflow.memoization INFO: Task 186 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/4a142d0b-6f45-4198-8977-93fc40cf470f_isrForCrosstalkSources_898385_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/4a142d0b-6f45-4198-8977-93fc40cf470f_isrForCrosstalkSources_898385_43.stderr
parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: <AppFuture at 0x1524c54980d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 627f950c2455264809e469a3f300a21a
parsl.dataflow.memoization INFO: Task 187 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0233d079-e93b-410e-9b8b-d5aedf7d1b72_isrForCrosstalkSources_898386_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0233d079-e93b-410e-9b8b-d5aedf7d1b72_isrForCrosstalkSources_898386_41.stderr
parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: <AppFuture at 0x1524c54a8b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 7deb2ae1db691f7e3d8fff933bc40b75
parsl.dataflow.memoization INFO: Task 188 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0e36c2e2-a32e-4f5b-8782-4a817a33c912_isrForCrosstalkSources_898386_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0e36c2e2-a32e-4f5b-8782-4a817a33c912_isrForCrosstalkSources_898386_36.stderr
parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: <AppFuture at 0x1524c54a8710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 189 has memoization hash cce1b182ef42eadcadf7324403aae849
parsl.dataflow.memoization INFO: Task 189 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/00ebeb8e-0850-45b2-9ce2-a1b99bcc9672_isrForCrosstalkSources_898282_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/00ebeb8e-0850-45b2-9ce2-a1b99bcc9672_isrForCrosstalkSources_898282_25.stderr
parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: <AppFuture at 0x1524c54a9c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 5f51f5724e9a210ee74759dbc453d7e6
parsl.dataflow.memoization INFO: Task 190 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/e91ffc52-338e-4338-8fd8-96835e4e6a0b_isrForCrosstalkSources_898282_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/e91ffc52-338e-4338-8fd8-96835e4e6a0b_isrForCrosstalkSources_898282_20.stderr
parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: <AppFuture at 0x1524c54aa7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 41ab9c629796a2c08fe942d5081dd80e
parsl.dataflow.memoization INFO: Task 191 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2eb96c39-96b6-4542-a3a1-ada8443ee1eb_isrForCrosstalkSources_898386_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2eb96c39-96b6-4542-a3a1-ada8443ee1eb_isrForCrosstalkSources_898386_2.stderr
parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: <AppFuture at 0x1524c54ab0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 7143f7ac3104f3f86a4ff497ccea382f
parsl.dataflow.memoization INFO: Task 192 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54853a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/a32810c9-d97f-467e-9dd7-ae0b12e5d7f3_isrForCrosstalkSources_898386_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/a32810c9-d97f-467e-9dd7-ae0b12e5d7f3_isrForCrosstalkSources_898386_24.stderr
parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: <AppFuture at 0x1524c54ab6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 2944c622f023f8226f5fce14add07361
parsl.dataflow.memoization INFO: Task 193 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/a1e6097a-da00-47a2-b5d3-f9424178eed5_isrForCrosstalkSources_898281_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/a1e6097a-da00-47a2-b5d3-f9424178eed5_isrForCrosstalkSources_898281_39.stderr
parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: <AppFuture at 0x1524c54abf90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 194 has memoization hash c62ae904dfa3db90efa086afd1500283
parsl.dataflow.memoization INFO: Task 194 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0d6d7d4f-da03-45f6-ab98-9535b5315c5c_isrForCrosstalkSources_898386_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0d6d7d4f-da03-45f6-ab98-9535b5315c5c_isrForCrosstalkSources_898386_51.stderr
parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: <AppFuture at 0x1524c5494650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 34734cee5fd2bdd6a2f401f504408840
parsl.dataflow.memoization INFO: Task 195 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/923197e3-4d32-4efa-aef9-de7736d8f41d_isrForCrosstalkSources_898385_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/923197e3-4d32-4efa-aef9-de7736d8f41d_isrForCrosstalkSources_898385_15.stderr
parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: <AppFuture at 0x1524c54b5ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 252340169dfa662843ab667174b30ad5
parsl.dataflow.memoization INFO: Task 196 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/1f6fa4f7-f087-4c34-9115-360ea4edea75_isrForCrosstalkSources_898387_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/1f6fa4f7-f087-4c34-9115-360ea4edea75_isrForCrosstalkSources_898387_38.stderr
parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: <AppFuture at 0x1524c54b59d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 35059e54ec43109cb6f54e273f6544da
parsl.dataflow.memoization INFO: Task 197 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54856c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/26bfa0aa-3550-4b78-aba1-afbfc3e7a052_isrForCrosstalkSources_898280_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/26bfa0aa-3550-4b78-aba1-afbfc3e7a052_isrForCrosstalkSources_898280_46.stderr
parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: <AppFuture at 0x1524c54b6090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 198 has memoization hash b313740c46ee2081d7606ceb01f1ec57
parsl.dataflow.memoization INFO: Task 198 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/94f62a8b-de51-4436-8290-d60b0408ca4d_isrForCrosstalkSources_898385_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/94f62a8b-de51-4436-8290-d60b0408ca4d_isrForCrosstalkSources_898385_55.stderr
parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: <AppFuture at 0x1524c54b6990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 0643190ea6be4cbe7f1032b6940dcf74
parsl.dataflow.memoization INFO: Task 199 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bb0f4037-073e-4caa-8183-f7adf46b46a5_isrForCrosstalkSources_898281_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bb0f4037-073e-4caa-8183-f7adf46b46a5_isrForCrosstalkSources_898281_8.stderr
parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: <AppFuture at 0x1524c54b7650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 200 has memoization hash fd87fb449f06754074b37b25e3249747
parsl.dataflow.memoization INFO: Task 200 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54858a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/a8ff940c-31ac-456e-b2c2-1323dff86095_isrForCrosstalkSources_898282_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/a8ff940c-31ac-456e-b2c2-1323dff86095_isrForCrosstalkSources_898282_32.stderr
parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: <AppFuture at 0x1524c54b6f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 201 has memoization hash a34415b88be3bcd2d8974a8e0f65bb3e
parsl.dataflow.memoization INFO: Task 201 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/00d086d7-9044-4680-b783-a13bc302a030_isrForCrosstalkSources_898386_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/00d086d7-9044-4680-b783-a13bc302a030_isrForCrosstalkSources_898386_56.stderr
parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: <AppFuture at 0x1524c54b7750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 1bd654a0f3731e00b487675e6a93419b
parsl.dataflow.memoization INFO: Task 202 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54859e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/db914e92-dab1-4410-90d6-3d938160d37b_isrForCrosstalkSources_898281_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/db914e92-dab1-4410-90d6-3d938160d37b_isrForCrosstalkSources_898281_25.stderr
parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: <AppFuture at 0x1524c54c0cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 6a8fc9c81555a484d01d00e1a96e67d0
parsl.dataflow.memoization INFO: Task 203 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/95bc3c06-7e98-43e6-9bb0-344f321744e3_isrForCrosstalkSources_898387_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/95bc3c06-7e98-43e6-9bb0-344f321744e3_isrForCrosstalkSources_898387_43.stderr
parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: <AppFuture at 0x1524c54c0090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 0d5276aca2bcc7e2a3ad865aa08b1a4d
parsl.dataflow.memoization INFO: Task 204 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/e086d7df-5ef8-4b4f-b828-18cf3c930989_isrForCrosstalkSources_898386_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/e086d7df-5ef8-4b4f-b828-18cf3c930989_isrForCrosstalkSources_898386_37.stderr
parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: <AppFuture at 0x1524c54c23d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 205 has memoization hash a28c15290355aa82aee6d6d88d52dab9
parsl.dataflow.memoization INFO: Task 205 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/f294daec-5f8c-4c96-b1e8-c83eb9a2f74f_isrForCrosstalkSources_898385_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/f294daec-5f8c-4c96-b1e8-c83eb9a2f74f_isrForCrosstalkSources_898385_20.stderr
parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: <AppFuture at 0x1524c54c1c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 4274bb769df40882a09890e78bce5e46
parsl.dataflow.memoization INFO: Task 206 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/669c7604-2662-488d-b798-ea29d7326fbe_isrForCrosstalkSources_898385_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/669c7604-2662-488d-b798-ea29d7326fbe_isrForCrosstalkSources_898385_21.stderr
parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: <AppFuture at 0x1524c54c3a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 117db7fe8de6c022c858d1883f88c467
parsl.dataflow.memoization INFO: Task 207 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/4df39916-772c-4bee-99f0-2fe63d6c2cc0_isrForCrosstalkSources_898280_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/4df39916-772c-4bee-99f0-2fe63d6c2cc0_isrForCrosstalkSources_898280_18.stderr
parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: <AppFuture at 0x1524c54c2dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 208 has memoization hash c7dff3a44ceeb00b5758138e1206e39a
parsl.dataflow.memoization INFO: Task 208 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7d1f7bba-b3cc-4bd4-ba1e-88c3c131bd8c_isrForCrosstalkSources_898385_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7d1f7bba-b3cc-4bd4-ba1e-88c3c131bd8c_isrForCrosstalkSources_898385_62.stderr
parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: <AppFuture at 0x1524c54c3b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 1b346318a3bed085baec58135b5f2b8e
parsl.dataflow.memoization INFO: Task 209 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ed51c2a9-4323-40b0-81a8-76a79b21269a_isrForCrosstalkSources_898387_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ed51c2a9-4323-40b0-81a8-76a79b21269a_isrForCrosstalkSources_898387_45.stderr
parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: <AppFuture at 0x1524c54c0ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 3e2705bcb174c171effa584eed57f463
parsl.dataflow.memoization INFO: Task 210 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/da95cc7e-ab7f-4b6b-901e-1044c46550cf_isrForCrosstalkSources_898385_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/da95cc7e-ab7f-4b6b-901e-1044c46550cf_isrForCrosstalkSources_898385_1.stderr
parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: <AppFuture at 0x1524c54c96d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 5d25d18079d9bc9ccefa14fda09d4c23
parsl.dataflow.memoization INFO: Task 211 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5485f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/df56b7fd-745a-4180-808d-b5ac5829402e_isrForCrosstalkSources_898387_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/df56b7fd-745a-4180-808d-b5ac5829402e_isrForCrosstalkSources_898387_55.stderr
parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: <AppFuture at 0x1524c54c97d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 1f955cf3e6e507686edeb8f3668d98a9
parsl.dataflow.memoization INFO: Task 212 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b83cecca-92d1-4d87-992d-771e4bf14809_isrForCrosstalkSources_898280_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b83cecca-92d1-4d87-992d-771e4bf14809_isrForCrosstalkSources_898280_11.stderr
parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: <AppFuture at 0x1524c54ca7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 769c87018074adb16b9f8ff09eeae225
parsl.dataflow.memoization INFO: Task 213 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/5e4c32b1-ec94-40d5-9231-67c70f112daf_isrForCrosstalkSources_898282_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/5e4c32b1-ec94-40d5-9231-67c70f112daf_isrForCrosstalkSources_898282_57.stderr
parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: <AppFuture at 0x1524c54ca910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 749ff4b1738c2b01cd5825fa97e8a8fe
parsl.dataflow.memoization INFO: Task 214 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54860c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d4e5a1e1-79a1-462c-bf70-c1669ad412e0_isrForCrosstalkSources_898387_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d4e5a1e1-79a1-462c-bf70-c1669ad412e0_isrForCrosstalkSources_898387_15.stderr
parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: <AppFuture at 0x1524c54cb9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 4a5e055c674c2655ede94bc097bc1560
parsl.dataflow.memoization INFO: Task 215 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4137ffe0-2a8f-48c8-94cf-32b231226632_isrForCrosstalkSources_898281_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4137ffe0-2a8f-48c8-94cf-32b231226632_isrForCrosstalkSources_898281_18.stderr
parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: <AppFuture at 0x1524c54cb710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 216 has memoization hash eef486e78931f905048dff572bfb0ada
parsl.dataflow.memoization INFO: Task 216 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54862a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/7cf81f20-46b4-4868-888a-a6bed80f5927_isrForCrosstalkSources_898282_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/7cf81f20-46b4-4868-888a-a6bed80f5927_isrForCrosstalkSources_898282_28.stderr
parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: <AppFuture at 0x1524c54cbc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 19f303d1f78cbf297803bbec942db52b
parsl.dataflow.memoization INFO: Task 217 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/16997706-3b1c-4ad6-9c12-cda7eb9bd5d9_isrForCrosstalkSources_898387_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/16997706-3b1c-4ad6-9c12-cda7eb9bd5d9_isrForCrosstalkSources_898387_61.stderr
parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: <AppFuture at 0x1524c54cbc10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 57d988e4a648094d04814ad04ea0d1bc
parsl.dataflow.memoization INFO: Task 218 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/916f3315-11f9-4040-b09c-442e4e869a6b_isrForCrosstalkSources_898282_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/916f3315-11f9-4040-b09c-442e4e869a6b_isrForCrosstalkSources_898282_26.stderr
parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: <AppFuture at 0x1524c54c5bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 12c81434deedf95d4493b8a4c1c36c4f
parsl.dataflow.memoization INFO: Task 219 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54863e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f13024d1-5ca9-4f2f-bb54-697258f87adb_isrForCrosstalkSources_898386_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f13024d1-5ca9-4f2f-bb54-697258f87adb_isrForCrosstalkSources_898386_40.stderr
parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: <AppFuture at 0x1524c54c55d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 874ea952fbde3d52b6f4ca5cf5b96f35
parsl.dataflow.memoization INFO: Task 220 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/8d1f44f1-62ae-4e3e-b3d3-0f8cc488da7d_isrForCrosstalkSources_898282_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/8d1f44f1-62ae-4e3e-b3d3-0f8cc488da7d_isrForCrosstalkSources_898282_30.stderr
parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: <AppFuture at 0x1524c54c6710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 45fe79ad83c8178b990ea66fd6e6b6a9
parsl.dataflow.memoization INFO: Task 221 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54865c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e69ed1de-f30b-4fcc-9b72-096f11075e00_isrForCrosstalkSources_898280_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e69ed1de-f30b-4fcc-9b72-096f11075e00_isrForCrosstalkSources_898280_39.stderr
parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: <AppFuture at 0x1524c54c6cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 67e05a211f6208ca7b54730248ad9527
parsl.dataflow.memoization INFO: Task 222 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/be32eff1-d676-4524-a200-50d58670734e_isrForCrosstalkSources_898387_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/be32eff1-d676-4524-a200-50d58670734e_isrForCrosstalkSources_898387_6.stderr
parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: <AppFuture at 0x1524c54c5dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 6feba2ef27e2057e154d08d523d90df4
parsl.dataflow.memoization INFO: Task 223 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54867a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/5019a8c2-2289-4139-89da-c2b2155b72c6_isrForCrosstalkSources_898280_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/5019a8c2-2289-4139-89da-c2b2155b72c6_isrForCrosstalkSources_898280_60.stderr
parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: <AppFuture at 0x1524c69dc490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 98e92fdb1cc3f363816ba8a928e52e59
parsl.dataflow.memoization INFO: Task 224 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b8949b8c-0a3b-4e1c-b30c-82a214de2394_isrForCrosstalkSources_898280_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b8949b8c-0a3b-4e1c-b30c-82a214de2394_isrForCrosstalkSources_898280_30.stderr
parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: <AppFuture at 0x1524c54c4fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 225 has memoization hash fc5d9c3defd10b1dbc79c163c969949a
parsl.dataflow.memoization INFO: Task 225 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2fd686ce-a8e3-4b53-aa3e-2dd3bd3431ff_isrForCrosstalkSources_898386_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2fd686ce-a8e3-4b53-aa3e-2dd3bd3431ff_isrForCrosstalkSources_898386_12.stderr
parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: <AppFuture at 0x1524c54b55d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 9e8f507270ad4764ae6ed236f8d2eaa1
parsl.dataflow.memoization INFO: Task 226 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54868e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/19605d16-05e9-4e1c-aff9-6afe3873ebc2_isrForCrosstalkSources_898280_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/19605d16-05e9-4e1c-aff9-6afe3873ebc2_isrForCrosstalkSources_898280_33.stderr
parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: <AppFuture at 0x1524c54dd6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 88cbdefd2c08cb3e8e5962e1a6c5175a
parsl.dataflow.memoization INFO: Task 227 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7f2222d3-ed87-4e48-a449-b742de417a56_isrForCrosstalkSources_898281_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7f2222d3-ed87-4e48-a449-b742de417a56_isrForCrosstalkSources_898281_24.stderr
parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: <AppFuture at 0x1524c54dd350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 68f0865dad9b690b3998019f125a925f
parsl.dataflow.memoization INFO: Task 228 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d9908ed3-02b9-4b0a-be4f-caac7cc7043b_isrForCrosstalkSources_898282_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d9908ed3-02b9-4b0a-be4f-caac7cc7043b_isrForCrosstalkSources_898282_19.stderr
parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: <AppFuture at 0x1524c54deed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 067e572cd996e4cb5dd63bf63b2b90a9
parsl.dataflow.memoization INFO: Task 229 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/eb6440f8-06fe-4ec1-93b5-68d8e8448682_isrForCrosstalkSources_898281_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/eb6440f8-06fe-4ec1-93b5-68d8e8448682_isrForCrosstalkSources_898281_1.stderr
parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: <AppFuture at 0x1524c54df350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 06281ffcc4c68cb96565492ae1823377
parsl.dataflow.memoization INFO: Task 230 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/58be7d9c-0b49-434c-a36b-0ce758c4125c_isrForCrosstalkSources_898386_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/58be7d9c-0b49-434c-a36b-0ce758c4125c_isrForCrosstalkSources_898386_49.stderr
parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: <AppFuture at 0x1524c54dff10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 3ca77446260ced1f19351ebc34dfd104
parsl.dataflow.memoization INFO: Task 231 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/c62ab383-1519-44c2-bcb0-707d43d30e94_isrForCrosstalkSources_898281_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/c62ab383-1519-44c2-bcb0-707d43d30e94_isrForCrosstalkSources_898281_17.stderr
parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: <AppFuture at 0x1524c54c0190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 53e7c423c09dbb50690dc0495385b2bb
parsl.dataflow.memoization INFO: Task 232 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b075fd2c-73db-4d85-a453-96bc1836e085_isrForCrosstalkSources_898282_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b075fd2c-73db-4d85-a453-96bc1836e085_isrForCrosstalkSources_898282_14.stderr
parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: <AppFuture at 0x1524c54e8d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 233 has memoization hash dbc93e0640874db9282e11e89d34b192
parsl.dataflow.memoization INFO: Task 233 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/f27d660e-c7e2-4c87-ad77-f174158bf067_isrForCrosstalkSources_898282_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/f27d660e-c7e2-4c87-ad77-f174158bf067_isrForCrosstalkSources_898282_2.stderr
parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: <AppFuture at 0x1524c54e9d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 234 has memoization hash bd2c57e5e1668c63ddf1d0517450e4f2
parsl.dataflow.memoization INFO: Task 234 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/663bb20b-86b1-4dc1-84e3-05dbe2387ebe_isrForCrosstalkSources_898387_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/663bb20b-86b1-4dc1-84e3-05dbe2387ebe_isrForCrosstalkSources_898387_25.stderr
parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: <AppFuture at 0x1524c54ea1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 150a3652f9748d73a5d1136575c5c129
parsl.dataflow.memoization INFO: Task 235 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4588621c-fefb-4ba5-a0c7-49580bf8d7ab_isrForCrosstalkSources_898282_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4588621c-fefb-4ba5-a0c7-49580bf8d7ab_isrForCrosstalkSources_898282_59.stderr
parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: <AppFuture at 0x1524c54ea510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 36f9252e96f69866a35cebae61fe572d
parsl.dataflow.memoization INFO: Task 236 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/57ca9658-be51-4b85-b760-71709da9e0b2_isrForCrosstalkSources_898281_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/57ca9658-be51-4b85-b760-71709da9e0b2_isrForCrosstalkSources_898281_45.stderr
parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: <AppFuture at 0x1524c54eb710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 2d303c848e6ddf0ce3540d3864176650
parsl.dataflow.memoization INFO: Task 237 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5486fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/3cff9b60-d63c-4b4b-824d-8973fbd7528b_isrForCrosstalkSources_898280_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/3cff9b60-d63c-4b4b-824d-8973fbd7528b_isrForCrosstalkSources_898280_26.stderr
parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: <AppFuture at 0x1524c54e9110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 85f4ac843a29bd3b427afd1902861aed
parsl.dataflow.memoization INFO: Task 238 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e98eb2e4-0b79-4bcd-b972-2a46ee7c1653_isrForCrosstalkSources_898281_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e98eb2e4-0b79-4bcd-b972-2a46ee7c1653_isrForCrosstalkSources_898281_19.stderr
parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: <AppFuture at 0x1524c54eb550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 239 has memoization hash a38d286654599146c5164ab7672a4b6b
parsl.dataflow.memoization INFO: Task 239 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/57032478-6179-40c1-a7f1-80d5f44bfed9_isrForCrosstalkSources_898282_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/57032478-6179-40c1-a7f1-80d5f44bfed9_isrForCrosstalkSources_898282_5.stderr
parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: <AppFuture at 0x1524c54ec950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 5eb724d0989e0573ac68e02473aed837
parsl.dataflow.memoization INFO: Task 240 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54871a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/732a4187-75c1-4773-b469-71c117fef941_isrForCrosstalkSources_898281_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/732a4187-75c1-4773-b469-71c117fef941_isrForCrosstalkSources_898281_48.stderr
parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: <AppFuture at 0x1524c54eda50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 4dd5db412a071d3993006d4c7cfba9d6
parsl.dataflow.memoization INFO: Task 241 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/aa9c5689-65c5-4930-995a-0e2837e8b351_isrForCrosstalkSources_898280_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/aa9c5689-65c5-4930-995a-0e2837e8b351_isrForCrosstalkSources_898280_8.stderr
parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: <AppFuture at 0x1524c54ece10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 1e6ab2f443879038aaba0a85bc961551
parsl.dataflow.memoization INFO: Task 242 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/40d56106-ebc6-4b6a-a4fa-ee5703c5f99c_isrForCrosstalkSources_898281_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/40d56106-ebc6-4b6a-a4fa-ee5703c5f99c_isrForCrosstalkSources_898281_36.stderr
parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: <AppFuture at 0x1524c54eed90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 243 has memoization hash db67900bb9fbc356940f35c3e225a5be
parsl.dataflow.memoization INFO: Task 243 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/fbf05147-d949-44ad-9904-f2ee06299757_isrForCrosstalkSources_898280_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/fbf05147-d949-44ad-9904-f2ee06299757_isrForCrosstalkSources_898280_45.stderr
parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: <AppFuture at 0x1524c54ee390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 244 has memoization hash f146db9415a7100fcb25f673da6e2044
parsl.dataflow.memoization INFO: Task 244 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/6c93c632-f7f1-4e01-9566-4de2a0132ce0_isrForCrosstalkSources_898280_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/6c93c632-f7f1-4e01-9566-4de2a0132ce0_isrForCrosstalkSources_898280_51.stderr
parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: <AppFuture at 0x1524c54ef210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 6bc44e9b5e2d65274d38a6237808ccb5
parsl.dataflow.memoization INFO: Task 245 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54874c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/de2c59f2-6a71-4928-ae14-0eee8c17b3a9_isrForCrosstalkSources_898280_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/de2c59f2-6a71-4928-ae14-0eee8c17b3a9_isrForCrosstalkSources_898280_32.stderr
parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: <AppFuture at 0x1524c54ef9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 246 has memoization hash b8e0d6c60deda890ed21d068ee939fe6
parsl.dataflow.memoization INFO: Task 246 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/62fb6b8b-829e-4169-85f4-06dcfcd4d9c3_isrForCrosstalkSources_898386_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/62fb6b8b-829e-4169-85f4-06dcfcd4d9c3_isrForCrosstalkSources_898386_42.stderr
parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: <AppFuture at 0x1524c54e8310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 5524f6c6cf8d3774e776ddae56ea9ee0
parsl.dataflow.memoization INFO: Task 247 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6e983895-8d1d-4efc-95b9-25ca103bcd29_isrForCrosstalkSources_898385_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6e983895-8d1d-4efc-95b9-25ca103bcd29_isrForCrosstalkSources_898385_16.stderr
parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: <AppFuture at 0x1524c54c10d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 1e3985acbce9aba00323d97f99927313
parsl.dataflow.memoization INFO: Task 248 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/45399a4f-e60a-4100-8e36-814ec5e52d16_isrForCrosstalkSources_898281_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/45399a4f-e60a-4100-8e36-814ec5e52d16_isrForCrosstalkSources_898281_27.stderr
parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: <AppFuture at 0x1524c54a9690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 91b59da9cb3778693119176ac8b622b3
parsl.dataflow.memoization INFO: Task 249 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54876a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bac37693-60dd-456e-bf2e-8a24de2ee18b_isrForCrosstalkSources_898281_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bac37693-60dd-456e-bf2e-8a24de2ee18b_isrForCrosstalkSources_898281_4.stderr
parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: <AppFuture at 0x1524c5501d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 250 has memoization hash cebe02c893df9bf483e5814b6fdc1817
parsl.dataflow.memoization INFO: Task 250 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54877e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/fd427fd4-c3f1-46c0-a981-14f079528eeb_isrForCrosstalkSources_898386_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/fd427fd4-c3f1-46c0-a981-14f079528eeb_isrForCrosstalkSources_898386_50.stderr
parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: <AppFuture at 0x1524c5502b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 251 has memoization hash f9ee17cadb7fc33840499c971b92688c
parsl.dataflow.memoization INFO: Task 251 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/75333039-d099-413d-8796-38a9baaefb29_isrForCrosstalkSources_898386_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/75333039-d099-413d-8796-38a9baaefb29_isrForCrosstalkSources_898386_43.stderr
parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: <AppFuture at 0x1524c5503010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 252 has memoization hash edb7dc15e4464da2b17f9556760f74ab
parsl.dataflow.memoization INFO: Task 252 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/13c3129a-4aa8-41f5-8c02-6d4ece4c6d86_isrForCrosstalkSources_898282_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/13c3129a-4aa8-41f5-8c02-6d4ece4c6d86_isrForCrosstalkSources_898282_13.stderr
parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: <AppFuture at 0x1524c5503790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 9db114d0c04316f0332efe25d4a0596f
parsl.dataflow.memoization INFO: Task 253 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/6d8317f5-9faf-4eaf-a708-f53def25bfa2_isrForCrosstalkSources_898282_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/6d8317f5-9faf-4eaf-a708-f53def25bfa2_isrForCrosstalkSources_898282_43.stderr
parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: <AppFuture at 0x1524c54eb750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 28ece017ad356b66eae687baa2695163
parsl.dataflow.memoization INFO: Task 254 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c54879c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bd83d7e2-017c-4901-93dd-e239ee4604ff_isrForCrosstalkSources_898282_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bd83d7e2-017c-4901-93dd-e239ee4604ff_isrForCrosstalkSources_898282_53.stderr
parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: <AppFuture at 0x1524c55089d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 255 has memoization hash ed5a1b69e11b015c837b53f04fb13c51
parsl.dataflow.memoization INFO: Task 255 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/25624f14-5d18-4fc0-93f1-eaa34ddd3f2d_isrForCrosstalkSources_898385_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/25624f14-5d18-4fc0-93f1-eaa34ddd3f2d_isrForCrosstalkSources_898385_2.stderr
parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: <AppFuture at 0x1524c5509050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 76e3399a8eac7c09a4f1560511ba6594
parsl.dataflow.memoization INFO: Task 256 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bdc212cb-d55f-483f-932c-c87ebe523b16_isrForCrosstalkSources_898281_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bdc212cb-d55f-483f-932c-c87ebe523b16_isrForCrosstalkSources_898281_22.stderr
parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: <AppFuture at 0x1524c55021d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 837a0ee800c66ebfe852b36cbbd4100d
parsl.dataflow.memoization INFO: Task 257 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e031157f-725f-4e41-adb6-4cb744f312fd_isrForCrosstalkSources_898280_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e031157f-725f-4e41-adb6-4cb744f312fd_isrForCrosstalkSources_898280_2.stderr
parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: <AppFuture at 0x1524c550ac50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 258 has memoization hash f0d148ffe53b155c138bc652136d1714
parsl.dataflow.memoization INFO: Task 258 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/5f0062c5-6e1a-435a-89a1-c224fc7d4a5a_isrForCrosstalkSources_898280_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/5f0062c5-6e1a-435a-89a1-c224fc7d4a5a_isrForCrosstalkSources_898280_36.stderr
parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: <AppFuture at 0x1524c550af90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 4c5eb7c59518dc55a47e114ddfd62aad
parsl.dataflow.memoization INFO: Task 259 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/dbfd9b4c-27d7-4496-a29f-1bff15ba286b_isrForCrosstalkSources_898386_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/dbfd9b4c-27d7-4496-a29f-1bff15ba286b_isrForCrosstalkSources_898386_54.stderr
parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: <AppFuture at 0x1524c550b150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 260 has memoization hash a8ebb6615f442e52d6b9d60ac8acfcfd
parsl.dataflow.memoization INFO: Task 260 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/26381f99-274e-4da1-937b-0fca039f42e3_isrForCrosstalkSources_898387_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/26381f99-274e-4da1-937b-0fca039f42e3_isrForCrosstalkSources_898387_62.stderr
parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: <AppFuture at 0x1524c550b550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 261 has memoization hash f01dce1690bbdf50b0ff3ef1bb5751ce
parsl.dataflow.memoization INFO: Task 261 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/097d3af5-2f19-4340-ab7b-e87d8f9e2b57_isrForCrosstalkSources_898386_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/097d3af5-2f19-4340-ab7b-e87d8f9e2b57_isrForCrosstalkSources_898386_47.stderr
parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: <AppFuture at 0x1524c55010d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 262 has memoization hash aabbf9c40104759a0bb140c4824f99c7
parsl.dataflow.memoization INFO: Task 262 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5487f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d3bd7965-655e-4993-9da4-62bf7985a08e_isrForCrosstalkSources_898387_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d3bd7965-655e-4993-9da4-62bf7985a08e_isrForCrosstalkSources_898387_16.stderr
parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: <AppFuture at 0x1524c5510a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 36cc30de8cc31d5351ff2361999b0d74
parsl.dataflow.memoization INFO: Task 263 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5484040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/01bd5c5d-c125-451f-8711-ce6da8560465_isrForCrosstalkSources_898280_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/01bd5c5d-c125-451f-8711-ce6da8560465_isrForCrosstalkSources_898280_3.stderr
parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: <AppFuture at 0x1524c5512110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 629a08d4e76d26278618826f569ec2b8
parsl.dataflow.memoization INFO: Task 264 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e1db5d57-058d-4dca-927e-5486c03cded6_isrForCrosstalkSources_898385_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e1db5d57-058d-4dca-927e-5486c03cded6_isrForCrosstalkSources_898385_37.stderr
parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: <AppFuture at 0x1524c5512350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 265 has memoization hash c080fca29df9f8befef7b878afdc2737
parsl.dataflow.memoization INFO: Task 265 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/ad232190-ed85-43fe-82f9-1b991387cd48_isrForCrosstalkSources_898386_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/ad232190-ed85-43fe-82f9-1b991387cd48_isrForCrosstalkSources_898386_53.stderr
parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: <AppFuture at 0x1524c5511a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 266 has memoization hash a737065167aaf209a7f8fc65dd669836
parsl.dataflow.memoization INFO: Task 266 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2136e3e6-f57b-4c94-baeb-3bd35c2c18c8_isrForCrosstalkSources_898281_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2136e3e6-f57b-4c94-baeb-3bd35c2c18c8_isrForCrosstalkSources_898281_47.stderr
parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: <AppFuture at 0x1524c5512ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 138be5193ca1679ad28ea1a9c0b17e75
parsl.dataflow.memoization INFO: Task 267 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55142c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2330cafa-715b-44e2-b9f9-ddd501f4ce5b_isrForCrosstalkSources_898282_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2330cafa-715b-44e2-b9f9-ddd501f4ce5b_isrForCrosstalkSources_898282_49.stderr
parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: <AppFuture at 0x1524c5512e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 1265d0b769f6b3851d47015ac6af91fe
parsl.dataflow.memoization INFO: Task 268 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/b066b15e-2b20-4d6b-a994-a3062c0af9fd_isrForCrosstalkSources_898281_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/b066b15e-2b20-4d6b-a994-a3062c0af9fd_isrForCrosstalkSources_898281_44.stderr
parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: <AppFuture at 0x1524c55133d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 269 has memoization hash d69769a7f711ee79b83be96235da1e62
parsl.dataflow.memoization INFO: Task 269 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/8fbc3b9d-263b-4d2a-a37c-a942f5280c0d_isrForCrosstalkSources_898385_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/8fbc3b9d-263b-4d2a-a37c-a942f5280c0d_isrForCrosstalkSources_898385_12.stderr
parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: <AppFuture at 0x1524c551cbd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 555cd136355fdff67b6a241c6b315f67
parsl.dataflow.memoization INFO: Task 270 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55144a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/52a8abee-fa66-43f2-9e52-6861d0de45df_isrForCrosstalkSources_898386_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/52a8abee-fa66-43f2-9e52-6861d0de45df_isrForCrosstalkSources_898386_44.stderr
parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: <AppFuture at 0x1524c54edcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 90d3fdfdf1dda6983163121e9fe41c10
parsl.dataflow.memoization INFO: Task 271 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d3b9d9cf-e01a-42fe-aca6-89e4422e8d64_isrForCrosstalkSources_898282_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d3b9d9cf-e01a-42fe-aca6-89e4422e8d64_isrForCrosstalkSources_898282_45.stderr
parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: <AppFuture at 0x1524c551dd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 70a9315cc61fa358628467c3bd5c8cd2
parsl.dataflow.memoization INFO: Task 272 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55145e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/96e341cb-9407-47ff-ae2b-c253577aa575_isrForCrosstalkSources_898281_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/96e341cb-9407-47ff-ae2b-c253577aa575_isrForCrosstalkSources_898281_40.stderr
parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: <AppFuture at 0x1524c551cf50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 270b821ec670a2794cecf43c9905d4d6
parsl.dataflow.memoization INFO: Task 273 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/87dfc4f7-0c68-4f58-9ab7-2e95540f72b4_isrForCrosstalkSources_898280_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/87dfc4f7-0c68-4f58-9ab7-2e95540f72b4_isrForCrosstalkSources_898280_44.stderr
parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: <AppFuture at 0x1524c551e850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 7dbeb46f2a44e768f20e4c07557a9bc7
parsl.dataflow.memoization INFO: Task 274 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/59a6ddc0-41a9-4976-b86f-3203d5ac31d1_isrForCrosstalkSources_898282_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/59a6ddc0-41a9-4976-b86f-3203d5ac31d1_isrForCrosstalkSources_898282_4.stderr
parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: <AppFuture at 0x1524c551fd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 275 has memoization hash c0459c292d75caf3db9643c201c34688
parsl.dataflow.memoization INFO: Task 275 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55147c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/0c9ad12a-090d-437d-a152-ae44fc6289ae_isrForCrosstalkSources_898385_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/0c9ad12a-090d-437d-a152-ae44fc6289ae_isrForCrosstalkSources_898385_47.stderr
parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: <AppFuture at 0x1524c551f990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 5c83dd016c5f0017203551bd6e3dfed8
parsl.dataflow.memoization INFO: Task 276 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/084b128f-2c05-4e60-bd63-79f281432299_isrForCrosstalkSources_898385_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/084b128f-2c05-4e60-bd63-79f281432299_isrForCrosstalkSources_898385_46.stderr
parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: <AppFuture at 0x1524c551f2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 37e09388f7e40f991ae3f8ae2a3dba22
parsl.dataflow.memoization INFO: Task 277 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/c19e8196-d555-40f5-afbc-96b7ca3845ef_isrForCrosstalkSources_898387_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/c19e8196-d555-40f5-afbc-96b7ca3845ef_isrForCrosstalkSources_898387_18.stderr
parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: <AppFuture at 0x1524c6ccd290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 4b5cd24dfabca221ef80414adb4d35b4
parsl.dataflow.memoization INFO: Task 278 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a9fb0f38-7430-4bb4-8273-39ba3a470804_isrForCrosstalkSources_898385_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a9fb0f38-7430-4bb4-8273-39ba3a470804_isrForCrosstalkSources_898385_27.stderr
parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: <AppFuture at 0x1524c552d390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 279 has memoization hash d97d9ca989717b593a392aa8a378c0b0
parsl.dataflow.memoization INFO: Task 279 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55149a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d3ffd0f0-2bf0-48f4-96e7-d2ddcda65dfb_isrForCrosstalkSources_898387_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d3ffd0f0-2bf0-48f4-96e7-d2ddcda65dfb_isrForCrosstalkSources_898387_47.stderr
parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: <AppFuture at 0x1524c552dad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 178eba76f83e649628e8c4b07aed4e2a
parsl.dataflow.memoization INFO: Task 280 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/320e9ff1-381c-4c33-896b-c77d52af6c34_isrForCrosstalkSources_898385_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/320e9ff1-381c-4c33-896b-c77d52af6c34_isrForCrosstalkSources_898385_19.stderr
parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: <AppFuture at 0x1524c552e190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 60b2a48e2c0476ac3752a40b8ed0570c
parsl.dataflow.memoization INFO: Task 281 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/bd381560-70ff-4804-a4b1-90c89eb46588_isrForCrosstalkSources_898385_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/bd381560-70ff-4804-a4b1-90c89eb46588_isrForCrosstalkSources_898385_5.stderr
parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: <AppFuture at 0x1524c552ec90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 282 has memoization hash aa476bd78f980e2789a86bbe0e4dc337
parsl.dataflow.memoization INFO: Task 282 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/72d7ac71-4984-45f6-820a-d2b6f07d2e1d_isrForCrosstalkSources_898387_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/72d7ac71-4984-45f6-820a-d2b6f07d2e1d_isrForCrosstalkSources_898387_37.stderr
parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: <AppFuture at 0x1524c552ec10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 283 has memoization hash a7c80e865e56ea7d8e7b7b0f41d41ec4
parsl.dataflow.memoization INFO: Task 283 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/b7019ecb-c53e-45f5-91e3-0de71317fe17_isrForCrosstalkSources_898387_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/b7019ecb-c53e-45f5-91e3-0de71317fe17_isrForCrosstalkSources_898387_24.stderr
parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: <AppFuture at 0x1524c552f350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 73c5a29ffd621ab701ebabae4edcb028
parsl.dataflow.memoization INFO: Task 284 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/cc18e4d9-45bd-4666-b7bc-915177711a8a_isrForCrosstalkSources_898282_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/cc18e4d9-45bd-4666-b7bc-915177711a8a_isrForCrosstalkSources_898282_34.stderr
parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: <AppFuture at 0x1524c5534110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 8232299b19256d73d30775b27b7c98a7
parsl.dataflow.memoization INFO: Task 285 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e5db69ad-dffd-4308-afdf-f7aebbb09d76_isrForCrosstalkSources_898385_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e5db69ad-dffd-4308-afdf-f7aebbb09d76_isrForCrosstalkSources_898385_38.stderr
parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: <AppFuture at 0x1524c5535350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 954f4e7e1820732168879fd3156c0e8f
parsl.dataflow.memoization INFO: Task 286 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ad76c485-a2f9-46b7-9eb2-5fca09128815_isrForCrosstalkSources_898387_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ad76c485-a2f9-46b7-9eb2-5fca09128815_isrForCrosstalkSources_898387_39.stderr
parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: <AppFuture at 0x1524c55358d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 8dd6da188df676f5bb53f4b27f02d668
parsl.dataflow.memoization INFO: Task 287 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/7b3ef90f-0801-45ac-8357-e8ba9e21a07d_isrForCrosstalkSources_898386_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/7b3ef90f-0801-45ac-8357-e8ba9e21a07d_isrForCrosstalkSources_898386_39.stderr
parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: <AppFuture at 0x1524c5536050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 996f5e09b7e1e35449f5f761ef6345c7
parsl.dataflow.memoization INFO: Task 288 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ce912925-2b06-4bff-bd0c-48682412c5e3_isrForCrosstalkSources_898387_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ce912925-2b06-4bff-bd0c-48682412c5e3_isrForCrosstalkSources_898387_17.stderr
parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: <AppFuture at 0x1524c5536910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 289 has memoization hash c6005b3614cdba985bdbbc7852abf54a
parsl.dataflow.memoization INFO: Task 289 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5514fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/db27546c-2e84-4157-ba38-811122d8fb21_isrForCrosstalkSources_898387_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/db27546c-2e84-4157-ba38-811122d8fb21_isrForCrosstalkSources_898387_11.stderr
parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: <AppFuture at 0x1524c5536cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 290 has memoization hash a6bb08490d142c92fad18c527ef7f3a3
parsl.dataflow.memoization INFO: Task 290 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/bcb09e7d-14af-421d-a018-b89e4d3154fa_isrForCrosstalkSources_898280_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/bcb09e7d-14af-421d-a018-b89e4d3154fa_isrForCrosstalkSources_898280_47.stderr
parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: <AppFuture at 0x1524c5537e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 291 has memoization hash f90bb08e73d1a2a833811b296e060521
parsl.dataflow.memoization INFO: Task 291 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55151c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/2c3e2fa6-d66a-44d8-a268-cdff0f67bf4b_isrForCrosstalkSources_898385_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/2c3e2fa6-d66a-44d8-a268-cdff0f67bf4b_isrForCrosstalkSources_898385_25.stderr
parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: <AppFuture at 0x1524c5510110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 0195518e8936a32ec2110a684cd6d523
parsl.dataflow.memoization INFO: Task 292 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c2ac3626-c307-4e7b-a9dc-c3649a62420e_isrForCrosstalkSources_898282_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c2ac3626-c307-4e7b-a9dc-c3649a62420e_isrForCrosstalkSources_898282_35.stderr
parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: <AppFuture at 0x1524c5540e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 293 has memoization hash ed6fea2e46724c738dbfafde6fc65f2f
parsl.dataflow.memoization INFO: Task 293 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55153a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/6c098e60-1588-4db7-bbaa-666abd9f184b_isrForCrosstalkSources_898280_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/6c098e60-1588-4db7-bbaa-666abd9f184b_isrForCrosstalkSources_898280_37.stderr
parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: <AppFuture at 0x1524c5541410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 6c9a50e24bdc27092ad1e453e429bdf7
parsl.dataflow.memoization INFO: Task 294 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e226cec5-918c-40f7-81b3-8a5372400631_isrForCrosstalkSources_898281_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e226cec5-918c-40f7-81b3-8a5372400631_isrForCrosstalkSources_898281_54.stderr
parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: <AppFuture at 0x1524c55419d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 47a10c478780851812436f1f1facb9e2
parsl.dataflow.memoization INFO: Task 295 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ef421e5c-8c8d-443b-8ece-9e1d87006410_isrForCrosstalkSources_898280_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ef421e5c-8c8d-443b-8ece-9e1d87006410_isrForCrosstalkSources_898280_17.stderr
parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: <AppFuture at 0x1524c55425d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 723fba16a9b22a4e0160551aff5e8c22
parsl.dataflow.memoization INFO: Task 296 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55154e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/81f6a94d-5235-4a76-b349-77f970520259_isrForCrosstalkSources_898386_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/81f6a94d-5235-4a76-b349-77f970520259_isrForCrosstalkSources_898386_52.stderr
parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: <AppFuture at 0x1524c5542a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 4e081cf1b700092b7b3d810b3000919d
parsl.dataflow.memoization INFO: Task 297 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/ebf0da5e-d4f5-4348-82d2-91fbc129d3b6_isrForCrosstalkSources_898281_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/ebf0da5e-d4f5-4348-82d2-91fbc129d3b6_isrForCrosstalkSources_898281_21.stderr
parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: <AppFuture at 0x1524c55436d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 62385d171b99f6ca57b6138c2f874499
parsl.dataflow.memoization INFO: Task 298 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55156c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2355ec33-bf8b-4f1d-ad59-3b5b3f73254f_isrForCrosstalkSources_898281_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2355ec33-bf8b-4f1d-ad59-3b5b3f73254f_isrForCrosstalkSources_898281_58.stderr
parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: <AppFuture at 0x1524c55435d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 299 has memoization hash c6957b27d2689400bb5b87f430ae0c9f
parsl.dataflow.memoization INFO: Task 299 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/db93e938-7210-42ae-b933-5fc6d97590a1_isrForCrosstalkSources_898386_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/db93e938-7210-42ae-b933-5fc6d97590a1_isrForCrosstalkSources_898386_9.stderr
parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: <AppFuture at 0x1524c554c950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 3592871d5b387151594e3461b065c94a
parsl.dataflow.memoization INFO: Task 300 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/362c4606-e4dd-4cde-bcb7-f25619c5e760_isrForCrosstalkSources_898385_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/362c4606-e4dd-4cde-bcb7-f25619c5e760_isrForCrosstalkSources_898385_18.stderr
parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: <AppFuture at 0x1524c554c810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 301 has memoization hash b96834f027a1e3babf2128a2993b960d
parsl.dataflow.memoization INFO: Task 301 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/578d3715-5c58-4eab-b24f-f17727e6b8f1_isrForCrosstalkSources_898281_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/578d3715-5c58-4eab-b24f-f17727e6b8f1_isrForCrosstalkSources_898281_61.stderr
parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: <AppFuture at 0x1524c54eded0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 302 has memoization hash d8029c3bbb38cbc1f42e680117babb5b
parsl.dataflow.memoization INFO: Task 302 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55158a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/95dc356a-5ef5-4914-a5d4-dc0188bbcda5_isrForCrosstalkSources_898387_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/95dc356a-5ef5-4914-a5d4-dc0188bbcda5_isrForCrosstalkSources_898387_23.stderr
parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: <AppFuture at 0x1524c554e650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 303 has memoization hash f17eb644fe17b8458419d3ed4b3d0669
parsl.dataflow.memoization INFO: Task 303 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55159e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/a105e266-a9d5-43f2-beb7-59fcd5a7a6b2_isrForCrosstalkSources_898387_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/a105e266-a9d5-43f2-beb7-59fcd5a7a6b2_isrForCrosstalkSources_898387_5.stderr
parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: <AppFuture at 0x1524c554e150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 304 has memoization hash e518120143357ccc1f68ab48611102c3
parsl.dataflow.memoization INFO: Task 304 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/40d68480-6578-401c-8afb-0714e6c84d7d_isrForCrosstalkSources_898386_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/40d68480-6578-401c-8afb-0714e6c84d7d_isrForCrosstalkSources_898386_38.stderr
parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: <AppFuture at 0x1524c554e210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 8c43b4db44ab22de8de7b96ab39b4d62
parsl.dataflow.memoization INFO: Task 305 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/3cef908e-c9f6-4633-8b4b-ed739fc5bcd6_isrForCrosstalkSources_898280_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/3cef908e-c9f6-4633-8b4b-ed739fc5bcd6_isrForCrosstalkSources_898280_61.stderr
parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: <AppFuture at 0x1524c554f8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 6d5437b5fc2d40215f4524850d7a5a4a
parsl.dataflow.memoization INFO: Task 306 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a5f72ee0-fe0c-4d12-8095-b7accdd905ca_isrForCrosstalkSources_898385_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a5f72ee0-fe0c-4d12-8095-b7accdd905ca_isrForCrosstalkSources_898385_7.stderr
parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: <AppFuture at 0x1524c54786d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 307 has memoization hash ef7e4b4c087b1ef35b9a3a9cf74b9153
parsl.dataflow.memoization INFO: Task 307 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/66d4d493-1b95-4161-aa48-5e2ad5cc8364_isrForCrosstalkSources_898386_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/66d4d493-1b95-4161-aa48-5e2ad5cc8364_isrForCrosstalkSources_898386_28.stderr
parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: <AppFuture at 0x1524c5540f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 10364d1fa1d23692e27f2b602ebcbab6
parsl.dataflow.memoization INFO: Task 308 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/5ca1e994-e8e3-411b-9492-161cbdd8bd9d_isrForCrosstalkSources_898281_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/5ca1e994-e8e3-411b-9492-161cbdd8bd9d_isrForCrosstalkSources_898281_38.stderr
parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: <AppFuture at 0x1524c5551390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 309 has memoization hash cb2f6eb61e03084d81fafd55b3a3d47b
parsl.dataflow.memoization INFO: Task 309 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/772d1418-66b3-4e80-a04b-da26afd7026e_isrForCrosstalkSources_898385_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/772d1418-66b3-4e80-a04b-da26afd7026e_isrForCrosstalkSources_898385_60.stderr
parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: <AppFuture at 0x1524c5551b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 8cba34eab67d28f6af7d136c0f80f25f
parsl.dataflow.memoization INFO: Task 310 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/33e9dc29-fe49-4fb7-8d60-2931032d48c9_isrForCrosstalkSources_898282_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/33e9dc29-fe49-4fb7-8d60-2931032d48c9_isrForCrosstalkSources_898282_1.stderr
parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: <AppFuture at 0x1524c5552710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 311 has memoization hash d418f68a373f4662f492c9aeea6105e8
parsl.dataflow.memoization INFO: Task 311 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/3568f671-d27b-4d32-bf75-7e692220f4ef_isrForCrosstalkSources_898282_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/3568f671-d27b-4d32-bf75-7e692220f4ef_isrForCrosstalkSources_898282_7.stderr
parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: <AppFuture at 0x1524c5552cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 6a094c5dd769a80917a6bd2eea3b5124
parsl.dataflow.memoization INFO: Task 312 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/65fb7dfd-3bee-40d8-93ec-b5ac8aeb07d0_isrForCrosstalkSources_898281_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/65fb7dfd-3bee-40d8-93ec-b5ac8aeb07d0_isrForCrosstalkSources_898281_55.stderr
parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: <AppFuture at 0x1524c55538d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 87a572efbc3b0a9fed04f6ccae29d2ed
parsl.dataflow.memoization INFO: Task 313 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/32c19c90-3a57-4f73-8217-0e24a5018307_isrForCrosstalkSources_898385_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/32c19c90-3a57-4f73-8217-0e24a5018307_isrForCrosstalkSources_898385_23.stderr
parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: <AppFuture at 0x1524c5543dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 8f11dd5124965f66b1b02c7718319812
parsl.dataflow.memoization INFO: Task 314 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5515f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/ce8c4e42-6dea-4ca4-9d4d-16663820195b_isrForCrosstalkSources_898386_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/ce8c4e42-6dea-4ca4-9d4d-16663820195b_isrForCrosstalkSources_898386_58.stderr
parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: <AppFuture at 0x1524c5360d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 315 has memoization hash f259640ad7e8f07375e6e3615b6cd650
parsl.dataflow.memoization INFO: Task 315 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55160c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b49ca4f0-9493-4359-aeb1-60d8478ad3c1_isrForCrosstalkSources_898280_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b49ca4f0-9493-4359-aeb1-60d8478ad3c1_isrForCrosstalkSources_898280_15.stderr
parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: <AppFuture at 0x1524c5360f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 53f1b7f796af8389d5db2eb3d6a1ac30
parsl.dataflow.memoization INFO: Task 316 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/5b4aa234-11c1-4e83-96d4-e0c163055295_isrForCrosstalkSources_898282_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/5b4aa234-11c1-4e83-96d4-e0c163055295_isrForCrosstalkSources_898282_18.stderr
parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: <AppFuture at 0x1524c5362510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 317 has memoization hash a71cfcc6d7f00279495249626a8220a9
parsl.dataflow.memoization INFO: Task 317 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d21fc3b7-8822-4f7b-9a2e-22fa43bdaaba_isrForCrosstalkSources_898282_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d21fc3b7-8822-4f7b-9a2e-22fa43bdaaba_isrForCrosstalkSources_898282_54.stderr
parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: <AppFuture at 0x1524c5362450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 5c1d8386b39509c16f0d25ecbb2a5fda
parsl.dataflow.memoization INFO: Task 318 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/c1d2c9ab-8113-47ce-866e-ee4b8405c02a_isrForCrosstalkSources_898387_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/c1d2c9ab-8113-47ce-866e-ee4b8405c02a_isrForCrosstalkSources_898387_31.stderr
parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: <AppFuture at 0x1524c5362e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 0f4801836c54f3fd23e8e3f815c6c4a0
parsl.dataflow.memoization INFO: Task 319 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55162a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/53ae2ec5-0291-48f2-9ad5-98415b27c72e_isrForCrosstalkSources_898281_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/53ae2ec5-0291-48f2-9ad5-98415b27c72e_isrForCrosstalkSources_898281_6.stderr
parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: <AppFuture at 0x1524c5363950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 576217ea7337789d599520895db87e9e
parsl.dataflow.memoization INFO: Task 320 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55163e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/607401a1-a2dc-4f23-9506-74cfa81125b3_isrForCrosstalkSources_898386_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/607401a1-a2dc-4f23-9506-74cfa81125b3_isrForCrosstalkSources_898386_8.stderr
parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: <AppFuture at 0x1524c5363bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 29bd78caa196412844b8150f3e6e629d
parsl.dataflow.memoization INFO: Task 321 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/751c9502-a4bf-479e-b516-5352c44e09d7_isrForCrosstalkSources_898385_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/751c9502-a4bf-479e-b516-5352c44e09d7_isrForCrosstalkSources_898385_41.stderr
parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: <AppFuture at 0x1524c53626d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 9868dd2b2a8b8e13520124d625c8d08e
parsl.dataflow.memoization INFO: Task 322 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6c8ae889-8547-4c3f-b7d7-85b13f8fb4d4_isrForCrosstalkSources_898385_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6c8ae889-8547-4c3f-b7d7-85b13f8fb4d4_isrForCrosstalkSources_898385_40.stderr
parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: <AppFuture at 0x1524c53689d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 1e9220faef8ae5573d4bc6bf51169175
parsl.dataflow.memoization INFO: Task 323 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/0200c459-0947-45b4-bd4f-9ac08da34147_isrForCrosstalkSources_898385_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/0200c459-0947-45b4-bd4f-9ac08da34147_isrForCrosstalkSources_898385_44.stderr
parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: <AppFuture at 0x1524c53692d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 324 has memoization hash be129f901e0490b0266f4a6b44c8a265
parsl.dataflow.memoization INFO: Task 324 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55165c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c4014a47-0d18-4c3e-b5b4-0c5b6c7a44cb_isrForCrosstalkSources_898282_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c4014a47-0d18-4c3e-b5b4-0c5b6c7a44cb_isrForCrosstalkSources_898282_55.stderr
parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: <AppFuture at 0x1524c5369bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 515d666e2887afb720e9d531f31c5741
parsl.dataflow.memoization INFO: Task 325 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/630ab270-f1f7-4257-b7a2-871b4701ec1c_isrForCrosstalkSources_898385_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/630ab270-f1f7-4257-b7a2-871b4701ec1c_isrForCrosstalkSources_898385_26.stderr
parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: <AppFuture at 0x1524c536a710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 1c69a12b9d3d4ceb34e0c807169f0144
parsl.dataflow.memoization INFO: Task 326 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55167a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4bbd3933-ff5c-4285-849b-6f0f97bd82b3_isrForCrosstalkSources_898281_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4bbd3933-ff5c-4285-849b-6f0f97bd82b3_isrForCrosstalkSources_898281_52.stderr
parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: <AppFuture at 0x1524c536b010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 327 has memoization hash e2d2547732936dcc33165c91c11a2fce
parsl.dataflow.memoization INFO: Task 327 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/e77856db-68f5-44b1-a1b2-855a5e9d112d_isrForCrosstalkSources_898387_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/e77856db-68f5-44b1-a1b2-855a5e9d112d_isrForCrosstalkSources_898387_27.stderr
parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: <AppFuture at 0x1524c536b690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 56cba9d7b023e9eeff7c90e6239dd3d2
parsl.dataflow.memoization INFO: Task 328 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/415f4bec-3d69-4046-90ac-2cb1ed6dfcbf_isrForCrosstalkSources_898386_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/415f4bec-3d69-4046-90ac-2cb1ed6dfcbf_isrForCrosstalkSources_898386_59.stderr
parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: <AppFuture at 0x1524c536ad90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 75775952474ca2810ad2db57bde11337
parsl.dataflow.memoization INFO: Task 329 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55168e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/628146a3-9b1b-403e-9f3f-3fdee99e6cf8_isrForCrosstalkSources_898386_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/628146a3-9b1b-403e-9f3f-3fdee99e6cf8_isrForCrosstalkSources_898386_55.stderr
parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: <AppFuture at 0x1524c536abd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 2cc21860577ef9a3dde3e11285520b36
parsl.dataflow.memoization INFO: Task 330 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/be6ac907-e2df-41f4-802a-fca91996f735_isrForCrosstalkSources_898385_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/be6ac907-e2df-41f4-802a-fca91996f735_isrForCrosstalkSources_898385_58.stderr
parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: <AppFuture at 0x1524c5380dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 37a3b264f5e34ae774945abc2e3e71f7
parsl.dataflow.memoization INFO: Task 331 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b32a9a92-4e2d-4d89-add2-88eea440b292_isrForCrosstalkSources_898282_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b32a9a92-4e2d-4d89-add2-88eea440b292_isrForCrosstalkSources_898282_62.stderr
parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: <AppFuture at 0x1524c5382710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 332 has memoization hash e30d6738a72a90318693aa7af2a32a5e
parsl.dataflow.memoization INFO: Task 332 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4393031c-16a2-4531-a913-3f19277e357a_isrForCrosstalkSources_898282_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4393031c-16a2-4531-a913-3f19277e357a_isrForCrosstalkSources_898282_15.stderr
parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: <AppFuture at 0x1524c53820d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 7e50ed825a4d122322335a7d79adecd6
parsl.dataflow.memoization INFO: Task 333 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/9918d6e0-2cc5-4c13-8e6c-57f79fec3663_isrForCrosstalkSources_898386_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/9918d6e0-2cc5-4c13-8e6c-57f79fec3663_isrForCrosstalkSources_898386_13.stderr
parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: <AppFuture at 0x1524c5382950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 6ec75560d0527e356175cb0830e23272
parsl.dataflow.memoization INFO: Task 334 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/4a28f157-388f-4f0d-92a0-1ae9ca750ea9_isrForCrosstalkSources_898280_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/4a28f157-388f-4f0d-92a0-1ae9ca750ea9_isrForCrosstalkSources_898280_10.stderr
parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: <AppFuture at 0x1524c5383010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 335 has memoization hash ccc36667a952f08f5b8b8dea3afc0044
parsl.dataflow.memoization INFO: Task 335 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/40a664ac-783b-4169-850f-1c2c48511bcf_isrForCrosstalkSources_898387_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/40a664ac-783b-4169-850f-1c2c48511bcf_isrForCrosstalkSources_898387_19.stderr
parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: <AppFuture at 0x1524c5383850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 336 has memoization hash d84fa0ab8216a63e06c61b5fd97a6e79
parsl.dataflow.memoization INFO: Task 336 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a5b11818-3de9-44b8-aa9c-04259c68e947_isrForCrosstalkSources_898385_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a5b11818-3de9-44b8-aa9c-04259c68e947_isrForCrosstalkSources_898385_17.stderr
parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: <AppFuture at 0x1524c5383390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 8459a0a91c7037567c341b8f0a31732c
parsl.dataflow.memoization INFO: Task 337 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/030be73a-4818-4936-b25c-8adf17812d7f_isrForCrosstalkSources_898387_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/030be73a-4818-4936-b25c-8adf17812d7f_isrForCrosstalkSources_898387_22.stderr
parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: <AppFuture at 0x1524c5388650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 338 has memoization hash b523b9ba150011ba7dacbfb90ac8cd2d
parsl.dataflow.memoization INFO: Task 338 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bab1cba7-112f-4d0b-86ab-2272f2372eaf_isrForCrosstalkSources_898282_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bab1cba7-112f-4d0b-86ab-2272f2372eaf_isrForCrosstalkSources_898282_9.stderr
parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: <AppFuture at 0x1524c5388310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 20d7caf24708403134fbd2baf2f103ec
parsl.dataflow.memoization INFO: Task 339 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5516f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/242a9ce4-d521-4ba1-a20f-d2a75e32523d_isrForCrosstalkSources_898386_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/242a9ce4-d521-4ba1-a20f-d2a75e32523d_isrForCrosstalkSources_898386_5.stderr
parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: <AppFuture at 0x1524c538a110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 2738386bff08c925be9c408936082ed7
parsl.dataflow.memoization INFO: Task 340 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4f98c16d-8a2c-4fb7-a6df-cffed4130194_isrForCrosstalkSources_898281_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4f98c16d-8a2c-4fb7-a6df-cffed4130194_isrForCrosstalkSources_898281_57.stderr
parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: <AppFuture at 0x1524c5389810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 82c93a12f44e621ce97e3afda5554e6b
parsl.dataflow.memoization INFO: Task 341 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/5ebf063f-31a2-4381-acef-ab567cfc6b42_isrForCrosstalkSources_898281_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/5ebf063f-31a2-4381-acef-ab567cfc6b42_isrForCrosstalkSources_898281_35.stderr
parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: <AppFuture at 0x1524c538af50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 0959e05bac64513bcbbb49b0bbe5052e
parsl.dataflow.memoization INFO: Task 342 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55171a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b190115a-6023-45d9-8086-f4d91f66f070_isrForCrosstalkSources_898280_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b190115a-6023-45d9-8086-f4d91f66f070_isrForCrosstalkSources_898280_19.stderr
parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: <AppFuture at 0x1524c538a990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 343 has memoization hash a7be1ddfde58798a19ce233177bf2638
parsl.dataflow.memoization INFO: Task 343 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55172e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/6c529368-b961-4b06-9a14-883c8585071a_isrForCrosstalkSources_898281_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/6c529368-b961-4b06-9a14-883c8585071a_isrForCrosstalkSources_898281_53.stderr
parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: <AppFuture at 0x1524c538bbd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 6e24ed6d767b77025cf5a7caba6f5110
parsl.dataflow.memoization INFO: Task 344 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1aa518c5-7761-426f-8c65-540f6ff37bf9_isrForCrosstalkSources_898282_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1aa518c5-7761-426f-8c65-540f6ff37bf9_isrForCrosstalkSources_898282_52.stderr
parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: <AppFuture at 0x1524c54ec410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 7f4d8a9c99507f208af38674db1af012
parsl.dataflow.memoization INFO: Task 345 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/720d663e-f23b-4a72-82cd-a5f9a6cda3ce_isrForCrosstalkSources_898387_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/720d663e-f23b-4a72-82cd-a5f9a6cda3ce_isrForCrosstalkSources_898387_1.stderr
parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: <AppFuture at 0x1524c538cf10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 346 has memoization hash bafbf13543f12a44874a6cfba97b2204
parsl.dataflow.memoization INFO: Task 346 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/027491eb-7356-44c6-843b-96a99238f955_isrForCrosstalkSources_898281_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/027491eb-7356-44c6-843b-96a99238f955_isrForCrosstalkSources_898281_2.stderr
parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: <AppFuture at 0x1524c538e450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 614af394068f163ef09c0fdeaf4948cf
parsl.dataflow.memoization INFO: Task 347 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55174c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3083d8ef-7457-446f-821d-f2a157e6ac07_isrForCrosstalkSources_898385_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3083d8ef-7457-446f-821d-f2a157e6ac07_isrForCrosstalkSources_898385_32.stderr
parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: <AppFuture at 0x1524c538e350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 74b0aaa2e07760e09868dd7800f3afde
parsl.dataflow.memoization INFO: Task 348 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/11304bca-6b73-4236-910e-1b2c68e4cdd9_isrForCrosstalkSources_898282_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/11304bca-6b73-4236-910e-1b2c68e4cdd9_isrForCrosstalkSources_898282_8.stderr
parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: <AppFuture at 0x1524c538db90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 0a0f72b4a1e951ab353972e0f175552f
parsl.dataflow.memoization INFO: Task 349 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/429c661b-6079-4f16-90b5-051fa2fdbed1_isrForCrosstalkSources_898387_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/429c661b-6079-4f16-90b5-051fa2fdbed1_isrForCrosstalkSources_898387_20.stderr
parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: <AppFuture at 0x1524c538fd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 87e6e3bdeddf6e18ce1953b4ff8c76ab
parsl.dataflow.memoization INFO: Task 350 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55176a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7226e17e-3565-4c14-a903-42eddc920ee3_isrForCrosstalkSources_898387_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7226e17e-3565-4c14-a903-42eddc920ee3_isrForCrosstalkSources_898387_28.stderr
parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: <AppFuture at 0x1524c538f6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 351 has memoization hash e39a0c32e43f9a31edcb06551d49988f
parsl.dataflow.memoization INFO: Task 351 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/34245fbc-5a96-4b96-ac81-bbac51cbcae3_isrForCrosstalkSources_898281_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/34245fbc-5a96-4b96-ac81-bbac51cbcae3_isrForCrosstalkSources_898281_37.stderr
parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: <AppFuture at 0x1524c53886d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 56f0d9bc164cb567197e1c2f7809ad61
parsl.dataflow.memoization INFO: Task 352 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55177e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ac712d0a-5231-4bb6-a624-86d8250609c4_isrForCrosstalkSources_898280_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ac712d0a-5231-4bb6-a624-86d8250609c4_isrForCrosstalkSources_898280_41.stderr
parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: <AppFuture at 0x1524c5398c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 353 has memoization hash f8b3542cf3a4605ebd9f8841d91483b7
parsl.dataflow.memoization INFO: Task 353 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b98fb9f3-7db6-4ded-979f-ab27ed74880c_isrForCrosstalkSources_898280_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b98fb9f3-7db6-4ded-979f-ab27ed74880c_isrForCrosstalkSources_898280_28.stderr
parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: <AppFuture at 0x1524c5398910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 6887d46c81304dbad9a27226bb7d832d
parsl.dataflow.memoization INFO: Task 354 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/aa6feb82-077d-4d2b-baf5-e9dfb6363e23_isrForCrosstalkSources_898282_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/aa6feb82-077d-4d2b-baf5-e9dfb6363e23_isrForCrosstalkSources_898282_56.stderr
parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: <AppFuture at 0x1524c539a990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 5c6893864d951a51c6767bd76bb8a651
parsl.dataflow.memoization INFO: Task 355 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55179c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b4d9aef6-3521-4525-8db0-81933032165f_isrForCrosstalkSources_898280_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b4d9aef6-3521-4525-8db0-81933032165f_isrForCrosstalkSources_898280_42.stderr
parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: <AppFuture at 0x1524c5398a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 0736e63cbd2ca6e9ef6ec68dfa31fc97
parsl.dataflow.memoization INFO: Task 356 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b55360d3-afee-4e6d-9752-28510fa8d426_isrForCrosstalkSources_898280_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b55360d3-afee-4e6d-9752-28510fa8d426_isrForCrosstalkSources_898280_4.stderr
parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: <AppFuture at 0x1524c5399ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 9a0ce51df469169ab60699fc2586173b
parsl.dataflow.memoization INFO: Task 357 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/07f758e8-ff48-47e5-8e81-39a531233093_isrForCrosstalkSources_898282_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/07f758e8-ff48-47e5-8e81-39a531233093_isrForCrosstalkSources_898282_44.stderr
parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: <AppFuture at 0x1524c539b810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 358 has memoization hash cd18f2619c4de304eec103ab173f6ecb
parsl.dataflow.memoization INFO: Task 358 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/2815bdd4-8a9f-4236-9d42-897deb77c605_isrForCrosstalkSources_898387_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/2815bdd4-8a9f-4236-9d42-897deb77c605_isrForCrosstalkSources_898387_29.stderr
parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: <AppFuture at 0x1524c539be90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 359 has memoization hash b252bf0be9e5af9db623cb4d72a6d32e
parsl.dataflow.memoization INFO: Task 359 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/37d1be8e-4765-41e1-9f27-047ede4b58ae_isrForCrosstalkSources_898281_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/37d1be8e-4765-41e1-9f27-047ede4b58ae_isrForCrosstalkSources_898281_5.stderr
parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: <AppFuture at 0x1524c53a0b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 345cb3fe5d8c65d87aa7fc75f156e20c
parsl.dataflow.memoization INFO: Task 360 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/558166c5-afdb-4318-844b-759f79186eef_isrForCrosstalkSources_898386_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/558166c5-afdb-4318-844b-759f79186eef_isrForCrosstalkSources_898386_27.stderr
parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: <AppFuture at 0x1524c53a1050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 361 has memoization hash cb0818ed866f942e7218457e7593dd40
parsl.dataflow.memoization INFO: Task 361 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c7154d1e-c429-4841-b43f-0c7b5ac48d99_isrForCrosstalkSources_898280_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c7154d1e-c429-4841-b43f-0c7b5ac48d99_isrForCrosstalkSources_898280_34.stderr
parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: <AppFuture at 0x1524c53a1ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 501f5a341f224040727570a37419fd69
parsl.dataflow.memoization INFO: Task 362 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7ebc477f-cfb6-401f-8e44-fa8955a3d6c4_isrForCrosstalkSources_898387_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7ebc477f-cfb6-401f-8e44-fa8955a3d6c4_isrForCrosstalkSources_898387_34.stderr
parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: <AppFuture at 0x1524c53a2450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 363 has memoization hash ae59b860807fe1a9f54e03404284e6ab
parsl.dataflow.memoization INFO: Task 363 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/42d6ed7f-4bae-45c4-819c-6b26fc9c39b1_isrForCrosstalkSources_898387_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/42d6ed7f-4bae-45c4-819c-6b26fc9c39b1_isrForCrosstalkSources_898387_41.stderr
parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: <AppFuture at 0x1524c53a2c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 4174c9352fe44ba278d8d08611455fc2
parsl.dataflow.memoization INFO: Task 364 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c5517ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c568fa43-2990-4607-a419-7e51d81abf78_isrForCrosstalkSources_898282_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c568fa43-2990-4607-a419-7e51d81abf78_isrForCrosstalkSources_898282_11.stderr
parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: <AppFuture at 0x1524c53a32d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 924e58243137bb669dcaebc287cf4242
parsl.dataflow.memoization INFO: Task 365 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c55140e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/89c839f5-00de-485a-9fa6-05a244de14ef_isrForCrosstalkSources_898386_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/89c839f5-00de-485a-9fa6-05a244de14ef_isrForCrosstalkSources_898386_6.stderr
parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: <AppFuture at 0x1524c53a3a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 9d893bcb42cce18929bf63858b5e3247
parsl.dataflow.memoization INFO: Task 366 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c53a80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/9a93525c-ce44-4f0d-845d-5293befd5f56_isrForCrosstalkSources_898280_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/9a93525c-ce44-4f0d-845d-5293befd5f56_isrForCrosstalkSources_898280_27.stderr
parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: <AppFuture at 0x1524c53a2050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 367 has memoization hash b559ba77febffa30329b6cf61b71774d
parsl.dataflow.memoization INFO: Task 367 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c53a8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/3b2f50cb-d647-489d-b83a-6e7e142be2ee_isrForCrosstalkSources_898386_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/3b2f50cb-d647-489d-b83a-6e7e142be2ee_isrForCrosstalkSources_898386_15.stderr
parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: <AppFuture at 0x1524c53b0b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 54c2c4da9b2fb5f7deb519f00efaa8d4
parsl.dataflow.memoization INFO: Task 368 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c53a82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7afb4b26-8028-4f5d-b669-295e7f2064e6_isrForCrosstalkSources_898281_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7afb4b26-8028-4f5d-b669-295e7f2064e6_isrForCrosstalkSources_898281_32.stderr
parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: <AppFuture at 0x1524c53b1350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 369 has memoization hash ee571742c7c94f3471948c33998072f6
parsl.dataflow.memoization INFO: Task 369 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c53a8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e174afc5-e8ae-47a8-9e09-df333ac75a7a_isrForCrosstalkSources_898280_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e174afc5-e8ae-47a8-9e09-df333ac75a7a_isrForCrosstalkSources_898280_58.stderr
parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: <AppFuture at 0x1524c53b2710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 6b5974477791d632c1f9850f12b828e9
parsl.dataflow.memoization INFO: Task 370 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c53a8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/06ca93a3-7940-480e-870f-24ba94a8ff84_isrForCrosstalkSources_898385_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/06ca93a3-7940-480e-870f-24ba94a8ff84_isrForCrosstalkSources_898385_54.stderr
parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: <AppFuture at 0x1524c53b1390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 641159561e594446662b413759c2dc6d
parsl.dataflow.memoization INFO: Task 371 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1524c53a8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/5918aaa5-c014-4721-bf6b-d623ce0496d7_isrForCrosstalkSources_898386_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/5918aaa5-c014-4721-bf6b-d623ce0496d7_isrForCrosstalkSources_898386_34.stderr
2024-03-11 23:00:29 proc_lsst.multi:146 [INFO]  found job 26021 in provider local
proc_lsst.multi INFO: found job 26021 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/0 running/pending blocks, and 0 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism
parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out
parsl.dataflow.strategy DEBUG: Requesting 92 more blocks
parsl.executors.status_handling INFO: Scaling out by 92 blocks
parsl.executors.status_handling INFO: Allocated block ID 1
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1
2024-03-11 23:00:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
2024-03-11 23:00:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-11 23:00:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17001157
proc_lsst.multi INFO: [multi] job_id 17001157
2024-03-11 23:00:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:29 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001157
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001157
parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17001157
parsl.executors.status_handling INFO: Allocated block ID 2
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2
2024-03-11 23:00:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
2024-03-11 23:00:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17001158
proc_lsst.multi INFO: [multi] job_id 17001158
2024-03-11 23:00:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-11 23:00:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:30 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001158
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001158
parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17001158
parsl.executors.status_handling INFO: Allocated block ID 3
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3
2024-03-11 23:00:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
2024-03-11 23:00:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-11 23:00:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17001159
proc_lsst.multi INFO: [multi] job_id 17001159
2024-03-11 23:00:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-11 23:00:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:30 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001159
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001159
parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17001159
parsl.executors.status_handling INFO: Allocated block ID 4
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4
2024-03-11 23:00:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
2024-03-11 23:00:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-11 23:00:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17001160
proc_lsst.multi INFO: [multi] job_id 17001160
2024-03-11 23:00:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-11 23:00:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:30 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001160
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001160
parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17001160
parsl.executors.status_handling INFO: Allocated block ID 5
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5
2024-03-11 23:00:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
2024-03-11 23:00:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-11 23:00:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:31 proc_lsst.multi:170 [INFO]  [multi] job_id 17001161
proc_lsst.multi INFO: [multi] job_id 17001161
2024-03-11 23:00:31 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:31 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:31 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001161
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001161
parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17001161
parsl.executors.status_handling INFO: Allocated block ID 6
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6
2024-03-11 23:00:31 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
2024-03-11 23:00:31 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:31 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:31 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:32 proc_lsst.multi:170 [INFO]  [multi] job_id 17001162
proc_lsst.multi INFO: [multi] job_id 17001162
2024-03-11 23:00:32 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-11 23:00:32 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:32 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001162
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001162
parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17001162
parsl.executors.status_handling INFO: Allocated block ID 7
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7
2024-03-11 23:00:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
2024-03-11 23:00:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:32 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-11 23:00:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:32 proc_lsst.multi:170 [INFO]  [multi] job_id 17001163
proc_lsst.multi INFO: [multi] job_id 17001163
2024-03-11 23:00:32 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-11 23:00:32 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:32 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001163
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001163
parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17001163
parsl.executors.status_handling INFO: Allocated block ID 8
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8
2024-03-11 23:00:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
2024-03-11 23:00:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:32 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-11 23:00:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:32 proc_lsst.multi:170 [INFO]  [multi] job_id 17001164
proc_lsst.multi INFO: [multi] job_id 17001164
2024-03-11 23:00:32 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-11 23:00:32 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:32 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001164
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001164
parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17001164
parsl.executors.status_handling INFO: Allocated block ID 9
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9
2024-03-11 23:00:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
2024-03-11 23:00:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:32 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-11 23:00:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:32 proc_lsst.multi:170 [INFO]  [multi] job_id 17001165
proc_lsst.multi INFO: [multi] job_id 17001165
2024-03-11 23:00:32 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-11 23:00:32 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:32 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001165
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001165
parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17001165
parsl.executors.status_handling INFO: Allocated block ID 10
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10
2024-03-11 23:00:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
2024-03-11 23:00:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:32 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-11 23:00:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:33 proc_lsst.multi:170 [INFO]  [multi] job_id 17001166
proc_lsst.multi INFO: [multi] job_id 17001166
2024-03-11 23:00:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-11 23:00:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:33 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001166
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001166
parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17001166
parsl.executors.status_handling INFO: Allocated block ID 11
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11
2024-03-11 23:00:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
2024-03-11 23:00:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-11 23:00:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:34 proc_lsst.multi:170 [INFO]  [multi] job_id 17001167
proc_lsst.multi INFO: [multi] job_id 17001167
2024-03-11 23:00:34 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-11 23:00:34 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:34 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001167
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001167
parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17001167
parsl.executors.status_handling INFO: Allocated block ID 12
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12
2024-03-11 23:00:34 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
2024-03-11 23:00:34 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:34 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-11 23:00:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:34 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:35 proc_lsst.multi:170 [INFO]  [multi] job_id 17001168
proc_lsst.multi INFO: [multi] job_id 17001168
2024-03-11 23:00:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-11 23:00:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:35 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001168
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001168
parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17001168
parsl.executors.status_handling INFO: Allocated block ID 13
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13
2024-03-11 23:00:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
2024-03-11 23:00:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-11 23:00:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:36 proc_lsst.multi:170 [INFO]  [multi] job_id 17001169
proc_lsst.multi INFO: [multi] job_id 17001169
2024-03-11 23:00:36 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-11 23:00:36 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:36 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001169
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001169
parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17001169
parsl.executors.status_handling INFO: Allocated block ID 14
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14
2024-03-11 23:00:36 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
2024-03-11 23:00:36 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:36 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:36 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-11 23:00:36 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:36 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17001170
proc_lsst.multi INFO: [multi] job_id 17001170
2024-03-11 23:00:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-11 23:00:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:37 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001170
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001170
parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17001170
parsl.executors.status_handling INFO: Allocated block ID 15
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15
2024-03-11 23:00:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
2024-03-11 23:00:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-11 23:00:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17001171
proc_lsst.multi INFO: [multi] job_id 17001171
2024-03-11 23:00:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-11 23:00:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:37 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001171
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001171
parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17001171
parsl.executors.status_handling INFO: Allocated block ID 16
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16
2024-03-11 23:00:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
2024-03-11 23:00:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-11 23:00:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:38 proc_lsst.multi:170 [INFO]  [multi] job_id 17001172
proc_lsst.multi INFO: [multi] job_id 17001172
2024-03-11 23:00:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-11 23:00:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:38 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001172
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001172
parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17001172
parsl.executors.status_handling INFO: Allocated block ID 17
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17
2024-03-11 23:00:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
2024-03-11 23:00:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-11 23:00:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17001173
proc_lsst.multi INFO: [multi] job_id 17001173
2024-03-11 23:00:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-11 23:00:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:39 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001173
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001173
parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17001173
parsl.executors.status_handling INFO: Allocated block ID 18
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18
2024-03-11 23:00:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
2024-03-11 23:00:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-11 23:00:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17001174
proc_lsst.multi INFO: [multi] job_id 17001174
2024-03-11 23:00:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-11 23:00:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:40 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001174
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001174
parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17001174
parsl.executors.status_handling INFO: Allocated block ID 19
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19
2024-03-11 23:00:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
2024-03-11 23:00:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-11 23:00:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17001175
proc_lsst.multi INFO: [multi] job_id 17001175
2024-03-11 23:00:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-11 23:00:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:41 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001175
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001175
parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17001175
parsl.executors.status_handling INFO: Allocated block ID 20
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20
2024-03-11 23:00:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
2024-03-11 23:00:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-11 23:00:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17001176
proc_lsst.multi INFO: [multi] job_id 17001176
2024-03-11 23:00:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:41 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001176
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001176
parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17001176
parsl.executors.status_handling INFO: Allocated block ID 21
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21
2024-03-11 23:00:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
2024-03-11 23:00:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17001177
proc_lsst.multi INFO: [multi] job_id 17001177
2024-03-11 23:00:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-11 23:00:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001177
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001177
parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17001177
parsl.executors.status_handling INFO: Allocated block ID 22
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22
2024-03-11 23:00:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
2024-03-11 23:00:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-11 23:00:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17001178
proc_lsst.multi INFO: [multi] job_id 17001178
2024-03-11 23:00:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-11 23:00:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001178
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001178
parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17001178
parsl.executors.status_handling INFO: Allocated block ID 23
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23
2024-03-11 23:00:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
2024-03-11 23:00:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-11 23:00:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17001179
proc_lsst.multi INFO: [multi] job_id 17001179
2024-03-11 23:00:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-11 23:00:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001179
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001179
parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17001179
parsl.executors.status_handling INFO: Allocated block ID 24
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24
2024-03-11 23:00:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
2024-03-11 23:00:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-11 23:00:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17001180
proc_lsst.multi INFO: [multi] job_id 17001180
2024-03-11 23:00:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-11 23:00:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:43 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001180
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001180
parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17001180
parsl.executors.status_handling INFO: Allocated block ID 25
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25
2024-03-11 23:00:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
2024-03-11 23:00:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-11 23:00:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17001181
proc_lsst.multi INFO: [multi] job_id 17001181
2024-03-11 23:00:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-11 23:00:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:43 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001181
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001181
parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17001181
parsl.executors.status_handling INFO: Allocated block ID 26
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26
2024-03-11 23:00:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
2024-03-11 23:00:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-11 23:00:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17001182
proc_lsst.multi INFO: [multi] job_id 17001182
2024-03-11 23:00:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-11 23:00:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:44 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001182
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001182
parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17001182
parsl.executors.status_handling INFO: Allocated block ID 27
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27
2024-03-11 23:00:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
2024-03-11 23:00:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-11 23:00:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17001183
proc_lsst.multi INFO: [multi] job_id 17001183
2024-03-11 23:00:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-11 23:00:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:44 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001183
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001183
parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17001183
parsl.executors.status_handling INFO: Allocated block ID 28
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28
2024-03-11 23:00:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
2024-03-11 23:00:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-11 23:00:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17001184
proc_lsst.multi INFO: [multi] job_id 17001184
2024-03-11 23:00:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-11 23:00:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:44 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001184
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001184
parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17001184
parsl.executors.status_handling INFO: Allocated block ID 29
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29
2024-03-11 23:00:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
2024-03-11 23:00:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-11 23:00:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:45 proc_lsst.multi:170 [INFO]  [multi] job_id 17001185
proc_lsst.multi INFO: [multi] job_id 17001185
2024-03-11 23:00:45 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-11 23:00:45 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:45 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001185
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001185
parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17001185
parsl.executors.status_handling INFO: Allocated block ID 30
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30
2024-03-11 23:00:45 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:45 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:45 proc_lsst.multi:170 [INFO]  [multi] job_id 17001186
proc_lsst.multi INFO: [multi] job_id 17001186
2024-03-11 23:00:45 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:45 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:45 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17001186
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17001186
parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17001186
parsl.executors.status_handling INFO: Allocated block ID 31
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31
2024-03-11 23:00:45 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:45 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:45 proc_lsst.multi:170 [INFO]  [multi] job_id 17001187
proc_lsst.multi INFO: [multi] job_id 17001187
2024-03-11 23:00:45 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:45 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:45 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17001187
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17001187
parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17001187
parsl.executors.status_handling INFO: Allocated block ID 32
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32
2024-03-11 23:00:45 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:45 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:45 proc_lsst.multi:170 [INFO]  [multi] job_id 17001188
proc_lsst.multi INFO: [multi] job_id 17001188
2024-03-11 23:00:45 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-11 23:00:45 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:45 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17001188
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17001188
parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17001188
parsl.executors.status_handling INFO: Allocated block ID 33
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33
2024-03-11 23:00:45 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:45 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:45 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-11 23:00:45 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:45 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11ca421e45978ad36a0ed58c6fe36072 with result from task 0
2024-03-11 23:00:46 proc_lsst.multi:170 [INFO]  [multi] job_id 17001189
proc_lsst.multi INFO: [multi] job_id 17001189
2024-03-11 23:00:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-11 23:00:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:46 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17001189
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17001189
parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17001189
parsl.executors.status_handling INFO: Allocated block ID 34
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34
2024-03-11 23:00:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow DEBUG: Task 0 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d1f8bf7b-6764-4331-913b-b056979dd644_isrForCrosstalkSources_898387_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d1f8bf7b-6764-4331-913b-b056979dd644_isrForCrosstalkSources_898387_44.stderr
parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a5ed114523d75c7fdd1a4b55f8da638d with result from task 1
parsl.dataflow.dflow DEBUG: Task 1 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/87ff1644-1a9a-463b-8fbb-db1aa991d608_isrForCrosstalkSources_898282_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/87ff1644-1a9a-463b-8fbb-db1aa991d608_isrForCrosstalkSources_898282_46.stderr
2024-03-11 23:00:46 proc_lsst.multi:170 [INFO]  [multi] job_id 17001190
proc_lsst.multi INFO: [multi] job_id 17001190
2024-03-11 23:00:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-11 23:00:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:46 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17001190
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17001190
parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17001190
parsl.executors.status_handling INFO: Allocated block ID 35
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35
2024-03-11 23:00:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:46 proc_lsst.multi:170 [INFO]  [multi] job_id 17001191
proc_lsst.multi INFO: [multi] job_id 17001191
2024-03-11 23:00:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:46 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17001191
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17001191
parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17001191
parsl.executors.status_handling INFO: Allocated block ID 36
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36
2024-03-11 23:00:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:46 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-11 23:00:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:47 proc_lsst.multi:170 [INFO]  [multi] job_id 17001192
proc_lsst.multi INFO: [multi] job_id 17001192
2024-03-11 23:00:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:47 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001192
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001192
parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17001192
parsl.executors.status_handling INFO: Allocated block ID 37
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37
2024-03-11 23:00:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:47 proc_lsst.multi:170 [INFO]  [multi] job_id 17001193
proc_lsst.multi INFO: [multi] job_id 17001193
2024-03-11 23:00:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-11 23:00:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:47 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001193
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001193
parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17001193
parsl.executors.status_handling INFO: Allocated block ID 38
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38
2024-03-11 23:00:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:47 proc_lsst.multi:170 [INFO]  [multi] job_id 17001194
proc_lsst.multi INFO: [multi] job_id 17001194
2024-03-11 23:00:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-11 23:00:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:47 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001194
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001194
parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17001194
parsl.executors.status_handling INFO: Allocated block ID 39
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39
2024-03-11 23:00:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:47 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-11 23:00:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17001195
proc_lsst.multi INFO: [multi] job_id 17001195
2024-03-11 23:00:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-11 23:00:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:48 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001195
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001195
parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17001195
parsl.executors.status_handling INFO: Allocated block ID 40
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40
2024-03-11 23:00:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17001196
proc_lsst.multi INFO: [multi] job_id 17001196
2024-03-11 23:00:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:48 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001196
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001196
parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17001196
parsl.executors.status_handling INFO: Allocated block ID 41
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41
2024-03-11 23:00:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17001197
proc_lsst.multi INFO: [multi] job_id 17001197
2024-03-11 23:00:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-11 23:00:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:48 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001197
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001197
parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17001197
parsl.executors.status_handling INFO: Allocated block ID 42
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42
2024-03-11 23:00:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17001198
proc_lsst.multi INFO: [multi] job_id 17001198
2024-03-11 23:00:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-11 23:00:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:48 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001198
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001198
parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17001198
parsl.executors.status_handling INFO: Allocated block ID 43
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43
2024-03-11 23:00:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:48 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-11 23:00:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:49 proc_lsst.multi:170 [INFO]  [multi] job_id 17001199
proc_lsst.multi INFO: [multi] job_id 17001199
2024-03-11 23:00:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-11 23:00:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:49 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001199
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001199
parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17001199
parsl.executors.status_handling INFO: Allocated block ID 44
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44
2024-03-11 23:00:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:49 proc_lsst.multi:170 [INFO]  [multi] job_id 17001200
proc_lsst.multi INFO: [multi] job_id 17001200
2024-03-11 23:00:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-11 23:00:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:49 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001200
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001200
parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17001200
parsl.executors.status_handling INFO: Allocated block ID 45
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45
2024-03-11 23:00:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:49 proc_lsst.multi:170 [INFO]  [multi] job_id 17001201
proc_lsst.multi INFO: [multi] job_id 17001201
2024-03-11 23:00:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-11 23:00:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:49 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001201
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001201
parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17001201
parsl.executors.status_handling INFO: Allocated block ID 46
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46
2024-03-11 23:00:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:49 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-11 23:00:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17001202
proc_lsst.multi INFO: [multi] job_id 17001202
2024-03-11 23:00:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-11 23:00:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001202
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001202
parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17001202
parsl.executors.status_handling INFO: Allocated block ID 47
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47
2024-03-11 23:00:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17001203
proc_lsst.multi INFO: [multi] job_id 17001203
2024-03-11 23:00:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-11 23:00:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001203
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001203
parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17001203
parsl.executors.status_handling INFO: Allocated block ID 48
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48
2024-03-11 23:00:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17001204
proc_lsst.multi INFO: [multi] job_id 17001204
2024-03-11 23:00:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-11 23:00:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001204
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001204
parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17001204
parsl.executors.status_handling INFO: Allocated block ID 49
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49
2024-03-11 23:00:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17001205
proc_lsst.multi INFO: [multi] job_id 17001205
2024-03-11 23:00:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-11 23:00:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001205
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001205
parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17001205
parsl.executors.status_handling INFO: Allocated block ID 50
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50
2024-03-11 23:00:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17001206
proc_lsst.multi INFO: [multi] job_id 17001206
2024-03-11 23:00:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-11 23:00:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001206
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001206
parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17001206
parsl.executors.status_handling INFO: Allocated block ID 51
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51
2024-03-11 23:00:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-11 23:00:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001207
proc_lsst.multi INFO: [multi] job_id 17001207
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001207
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001207
parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17001207
parsl.executors.status_handling INFO: Allocated block ID 52
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001208
proc_lsst.multi INFO: [multi] job_id 17001208
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001208
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001208
parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17001208
parsl.executors.status_handling INFO: Allocated block ID 53
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001209
proc_lsst.multi INFO: [multi] job_id 17001209
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001209
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001209
parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17001209
parsl.executors.status_handling INFO: Allocated block ID 54
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001210
proc_lsst.multi INFO: [multi] job_id 17001210
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001210
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001210
parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17001210
parsl.executors.status_handling INFO: Allocated block ID 55
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001211
proc_lsst.multi INFO: [multi] job_id 17001211
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17001211
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17001211
parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17001211
parsl.executors.status_handling INFO: Allocated block ID 56
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001212
proc_lsst.multi INFO: [multi] job_id 17001212
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001212
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001212
parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17001212
parsl.executors.status_handling INFO: Allocated block ID 57
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001213
proc_lsst.multi INFO: [multi] job_id 17001213
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001213
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001213
parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17001213
parsl.executors.status_handling INFO: Allocated block ID 58
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001214
proc_lsst.multi INFO: [multi] job_id 17001214
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001214
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001214
parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17001214
parsl.executors.status_handling INFO: Allocated block ID 59
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001215
proc_lsst.multi INFO: [multi] job_id 17001215
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001215
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001215
parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17001215
parsl.executors.status_handling INFO: Allocated block ID 60
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17001216
proc_lsst.multi INFO: [multi] job_id 17001216
2024-03-11 23:00:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001216
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001216
parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17001216
parsl.executors.status_handling INFO: Allocated block ID 61
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61
2024-03-11 23:00:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:51 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:52 proc_lsst.multi:170 [INFO]  [multi] job_id 17001217
proc_lsst.multi INFO: [multi] job_id 17001217
2024-03-11 23:00:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-11 23:00:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:52 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001217
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001217
parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17001217
parsl.executors.status_handling INFO: Allocated block ID 62
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62
2024-03-11 23:00:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:52 proc_lsst.multi:170 [INFO]  [multi] job_id 17001218
proc_lsst.multi INFO: [multi] job_id 17001218
2024-03-11 23:00:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-11 23:00:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:52 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001218
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001218
parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17001218
parsl.executors.status_handling INFO: Allocated block ID 63
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63
2024-03-11 23:00:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:52 proc_lsst.multi:170 [INFO]  [multi] job_id 17001219
proc_lsst.multi INFO: [multi] job_id 17001219
2024-03-11 23:00:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-11 23:00:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:52 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001219
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001219
parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17001219
parsl.executors.status_handling INFO: Allocated block ID 64
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64
2024-03-11 23:00:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:52 proc_lsst.multi:170 [INFO]  [multi] job_id 17001220
proc_lsst.multi INFO: [multi] job_id 17001220
2024-03-11 23:00:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-11 23:00:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:52 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001220
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001220
parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17001220
parsl.executors.status_handling INFO: Allocated block ID 65
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65
2024-03-11 23:00:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:52 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-11 23:00:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:53 proc_lsst.multi:170 [INFO]  [multi] job_id 17001221
proc_lsst.multi INFO: [multi] job_id 17001221
2024-03-11 23:00:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-11 23:00:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:53 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001221
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001221
parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17001221
parsl.executors.status_handling INFO: Allocated block ID 66
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66
2024-03-11 23:00:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:53 proc_lsst.multi:170 [INFO]  [multi] job_id 17001222
proc_lsst.multi INFO: [multi] job_id 17001222
2024-03-11 23:00:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-11 23:00:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:53 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001222
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001222
parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17001222
parsl.executors.status_handling INFO: Allocated block ID 67
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67
2024-03-11 23:00:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:53 proc_lsst.multi:170 [INFO]  [multi] job_id 17001223
proc_lsst.multi INFO: [multi] job_id 17001223
2024-03-11 23:00:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-11 23:00:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:53 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001223
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001223
parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17001223
parsl.executors.status_handling INFO: Allocated block ID 68
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68
2024-03-11 23:00:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:53 proc_lsst.multi:170 [INFO]  [multi] job_id 17001224
proc_lsst.multi INFO: [multi] job_id 17001224
2024-03-11 23:00:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-11 23:00:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:53 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001224
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001224
parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17001224
parsl.executors.status_handling INFO: Allocated block ID 69
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69
2024-03-11 23:00:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:53 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-11 23:00:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:54 proc_lsst.multi:170 [INFO]  [multi] job_id 17001225
proc_lsst.multi INFO: [multi] job_id 17001225
2024-03-11 23:00:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-11 23:00:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:54 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001225
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001225
parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17001225
parsl.executors.status_handling INFO: Allocated block ID 70
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70
2024-03-11 23:00:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:54 proc_lsst.multi:170 [INFO]  [multi] job_id 17001226
proc_lsst.multi INFO: [multi] job_id 17001226
2024-03-11 23:00:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-11 23:00:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:54 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001226
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001226
parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17001226
parsl.executors.status_handling INFO: Allocated block ID 71
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71
2024-03-11 23:00:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:54 proc_lsst.multi:170 [INFO]  [multi] job_id 17001227
proc_lsst.multi INFO: [multi] job_id 17001227
2024-03-11 23:00:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-11 23:00:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:54 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001227
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001227
parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17001227
parsl.executors.status_handling INFO: Allocated block ID 72
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72
2024-03-11 23:00:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:54 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-11 23:00:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:55 proc_lsst.multi:170 [INFO]  [multi] job_id 17001228
proc_lsst.multi INFO: [multi] job_id 17001228
2024-03-11 23:00:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-11 23:00:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001228
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001228
parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17001228
parsl.executors.status_handling INFO: Allocated block ID 73
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73
2024-03-11 23:00:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:55 proc_lsst.multi:170 [INFO]  [multi] job_id 17001229
proc_lsst.multi INFO: [multi] job_id 17001229
2024-03-11 23:00:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-11 23:00:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001229
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001229
parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17001229
parsl.executors.status_handling INFO: Allocated block ID 74
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74
2024-03-11 23:00:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:55 proc_lsst.multi:170 [INFO]  [multi] job_id 17001230
proc_lsst.multi INFO: [multi] job_id 17001230
2024-03-11 23:00:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-11 23:00:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001230
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001230
parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17001230
parsl.executors.status_handling INFO: Allocated block ID 75
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75
2024-03-11 23:00:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:55 proc_lsst.multi:170 [INFO]  [multi] job_id 17001231
proc_lsst.multi INFO: [multi] job_id 17001231
2024-03-11 23:00:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001231
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001231
parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17001231
parsl.executors.status_handling INFO: Allocated block ID 76
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76
2024-03-11 23:00:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:55 proc_lsst.multi:170 [INFO]  [multi] job_id 17001232
proc_lsst.multi INFO: [multi] job_id 17001232
2024-03-11 23:00:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-11 23:00:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001232
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001232
parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17001232
parsl.executors.status_handling INFO: Allocated block ID 77
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77
2024-03-11 23:00:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:55 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-11 23:00:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:56 proc_lsst.multi:170 [INFO]  [multi] job_id 17001233
proc_lsst.multi INFO: [multi] job_id 17001233
2024-03-11 23:00:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-11 23:00:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:56 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001233
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001233
parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17001233
parsl.executors.status_handling INFO: Allocated block ID 78
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78
2024-03-11 23:00:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:56 proc_lsst.multi:170 [INFO]  [multi] job_id 17001234
proc_lsst.multi INFO: [multi] job_id 17001234
2024-03-11 23:00:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-11 23:00:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:56 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001234
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001234
parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17001234
parsl.executors.status_handling INFO: Allocated block ID 79
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79
2024-03-11 23:00:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9cd0b5350c2d4ed1e7b75042759ebfe3 with result from task 3
parsl.dataflow.dflow DEBUG: Task 3 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/120e993b-91cd-4a61-a610-ccbd8db53cfa_isrForCrosstalkSources_898280_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/120e993b-91cd-4a61-a610-ccbd8db53cfa_isrForCrosstalkSources_898280_12.stderr
2024-03-11 23:00:56 proc_lsst.multi:170 [INFO]  [multi] job_id 17001235
proc_lsst.multi INFO: [multi] job_id 17001235
2024-03-11 23:00:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-11 23:00:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:56 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001235
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001235
parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17001235
parsl.executors.status_handling INFO: Allocated block ID 80
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80
2024-03-11 23:00:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:56 proc_lsst.multi:170 [INFO]  [multi] job_id 17001236
proc_lsst.multi INFO: [multi] job_id 17001236
2024-03-11 23:00:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-11 23:00:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:56 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001236
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001236
parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17001236
parsl.executors.status_handling INFO: Allocated block ID 81
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81
2024-03-11 23:00:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:56 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-11 23:00:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:57 proc_lsst.multi:170 [INFO]  [multi] job_id 17001237
proc_lsst.multi INFO: [multi] job_id 17001237
2024-03-11 23:00:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-11 23:00:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:57 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001237
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001237
parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17001237
parsl.executors.status_handling INFO: Allocated block ID 82
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82
2024-03-11 23:00:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ce01a0f9ad153380448a357028d266d with result from task 2
2024-03-11 23:00:57 proc_lsst.multi:170 [INFO]  [multi] job_id 17001238
proc_lsst.multi INFO: [multi] job_id 17001238
2024-03-11 23:00:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-11 23:00:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:57 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001238
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001238
parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17001238
parsl.executors.status_handling INFO: Allocated block ID 83
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83
2024-03-11 23:00:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow DEBUG: Task 2 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2dc7cf98-e7d2-4d09-b3cc-7cdf8ec7efb0_isrForCrosstalkSources_898281_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2dc7cf98-e7d2-4d09-b3cc-7cdf8ec7efb0_isrForCrosstalkSources_898281_42.stderr
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:57 proc_lsst.multi:170 [INFO]  [multi] job_id 17001240
proc_lsst.multi INFO: [multi] job_id 17001240
2024-03-11 23:00:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-11 23:00:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:57 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001240
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001240
parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17001240
parsl.executors.status_handling INFO: Allocated block ID 84
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84
2024-03-11 23:00:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:57 proc_lsst.multi:170 [INFO]  [multi] job_id 17001241
proc_lsst.multi INFO: [multi] job_id 17001241
2024-03-11 23:00:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-11 23:00:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:57 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001241
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001241
parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17001241
parsl.executors.status_handling INFO: Allocated block ID 85
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85
2024-03-11 23:00:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:57 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-11 23:00:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:58 proc_lsst.multi:170 [INFO]  [multi] job_id 17001242
proc_lsst.multi INFO: [multi] job_id 17001242
2024-03-11 23:00:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:58 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001242
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001242
parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17001242
parsl.executors.status_handling INFO: Allocated block ID 86
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86
2024-03-11 23:00:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:58 proc_lsst.multi:170 [INFO]  [multi] job_id 17001243
proc_lsst.multi INFO: [multi] job_id 17001243
2024-03-11 23:00:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-11 23:00:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:58 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001243
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001243
parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17001243
parsl.executors.status_handling INFO: Allocated block ID 87
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87
2024-03-11 23:00:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:58 proc_lsst.multi:170 [INFO]  [multi] job_id 17001244
proc_lsst.multi INFO: [multi] job_id 17001244
2024-03-11 23:00:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-11 23:00:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:58 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001244
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001244
parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17001244
parsl.executors.status_handling INFO: Allocated block ID 88
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88
2024-03-11 23:00:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:58 proc_lsst.multi:170 [INFO]  [multi] job_id 17001245
proc_lsst.multi INFO: [multi] job_id 17001245
2024-03-11 23:00:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-11 23:00:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:58 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001245
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001245
parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17001245
parsl.executors.status_handling INFO: Allocated block ID 89
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89
2024-03-11 23:00:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:58 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-11 23:00:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17001246
proc_lsst.multi INFO: [multi] job_id 17001246
2024-03-11 23:00:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-11 23:00:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001246
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001246
parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17001246
parsl.executors.status_handling INFO: Allocated block ID 90
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90
2024-03-11 23:00:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17001247
proc_lsst.multi INFO: [multi] job_id 17001247
2024-03-11 23:00:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-11 23:00:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001247
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001247
parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17001247
parsl.executors.status_handling INFO: Allocated block ID 91
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91
2024-03-11 23:00:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17001248
proc_lsst.multi INFO: [multi] job_id 17001248
2024-03-11 23:00:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-11 23:00:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001248
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001248
parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17001248
parsl.executors.status_handling INFO: Allocated block ID 92
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92
2024-03-11 23:00:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-11 23:00:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-11 23:00:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-11 23:00:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,n3010,10.64.129.10,127.0.0.1,198.48.92.26,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54358 --result_port=54329 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-11 23:00:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17001249
proc_lsst.multi INFO: [multi] job_id 17001249
2024-03-11 23:00:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 37
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37
2024-03-11 23:00:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-11 23:00:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17001249
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17001249
parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17001249
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
2024-03-11 23:00:59 proc_lsst.multi:146 [INFO]  found job 26021 in provider local
proc_lsst.multi INFO: found job 26021 in provider local
2024-03-11 23:01:00 proc_lsst.multi:146 [INFO]  found job 17001157 in provider astro
proc_lsst.multi INFO: found job 17001157 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:00 proc_lsst.multi:146 [INFO]  found job 17001158 in provider astro
proc_lsst.multi INFO: found job 17001158 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:00 proc_lsst.multi:146 [INFO]  found job 17001159 in provider astro
proc_lsst.multi INFO: found job 17001159 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:01 proc_lsst.multi:146 [INFO]  found job 17001160 in provider astro
proc_lsst.multi INFO: found job 17001160 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:01 proc_lsst.multi:146 [INFO]  found job 17001161 in provider astro
proc_lsst.multi INFO: found job 17001161 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:01 proc_lsst.multi:146 [INFO]  found job 17001162 in provider astro
proc_lsst.multi INFO: found job 17001162 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:01 proc_lsst.multi:146 [INFO]  found job 17001163 in provider astro
proc_lsst.multi INFO: found job 17001163 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:01 proc_lsst.multi:146 [INFO]  found job 17001164 in provider astro
proc_lsst.multi INFO: found job 17001164 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:02 proc_lsst.multi:146 [INFO]  found job 17001165 in provider astro
proc_lsst.multi INFO: found job 17001165 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:02 proc_lsst.multi:146 [INFO]  found job 17001166 in provider astro
proc_lsst.multi INFO: found job 17001166 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:02 proc_lsst.multi:146 [INFO]  found job 17001167 in provider astro
proc_lsst.multi INFO: found job 17001167 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:02 proc_lsst.multi:146 [INFO]  found job 17001168 in provider astro
proc_lsst.multi INFO: found job 17001168 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:02 proc_lsst.multi:146 [INFO]  found job 17001169 in provider astro
proc_lsst.multi INFO: found job 17001169 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:02 proc_lsst.multi:146 [INFO]  found job 17001170 in provider astro
proc_lsst.multi INFO: found job 17001170 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:02 proc_lsst.multi:146 [INFO]  found job 17001171 in provider astro
proc_lsst.multi INFO: found job 17001171 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:03 proc_lsst.multi:146 [INFO]  found job 17001172 in provider astro
proc_lsst.multi INFO: found job 17001172 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:03 proc_lsst.multi:146 [INFO]  found job 17001173 in provider astro
proc_lsst.multi INFO: found job 17001173 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:03 proc_lsst.multi:146 [INFO]  found job 17001174 in provider astro
proc_lsst.multi INFO: found job 17001174 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:04 proc_lsst.multi:146 [INFO]  found job 17001175 in provider astro
proc_lsst.multi INFO: found job 17001175 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:04 proc_lsst.multi:146 [INFO]  found job 17001176 in provider astro
proc_lsst.multi INFO: found job 17001176 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:05 proc_lsst.multi:146 [INFO]  found job 17001177 in provider astro
proc_lsst.multi INFO: found job 17001177 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:05 proc_lsst.multi:146 [INFO]  found job 17001178 in provider astro
proc_lsst.multi INFO: found job 17001178 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:05 proc_lsst.multi:146 [INFO]  found job 17001179 in provider astro
proc_lsst.multi INFO: found job 17001179 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:05 proc_lsst.multi:146 [INFO]  found job 17001180 in provider astro
proc_lsst.multi INFO: found job 17001180 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:05 proc_lsst.multi:146 [INFO]  found job 17001181 in provider astro
proc_lsst.multi INFO: found job 17001181 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:06 proc_lsst.multi:146 [INFO]  found job 17001182 in provider astro
proc_lsst.multi INFO: found job 17001182 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:06 proc_lsst.multi:146 [INFO]  found job 17001183 in provider astro
proc_lsst.multi INFO: found job 17001183 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 086dc1ccf9ca50359f02f734ed98a834 with result from task 4
parsl.dataflow.dflow DEBUG: Task 4 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d4e3616b-dedb-4c8d-ae1c-1504d7d33b77_isrForCrosstalkSources_898282_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d4e3616b-dedb-4c8d-ae1c-1504d7d33b77_isrForCrosstalkSources_898282_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:06 proc_lsst.multi:146 [INFO]  found job 17001184 in provider astro
proc_lsst.multi INFO: found job 17001184 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:07 proc_lsst.multi:146 [INFO]  found job 17001185 in provider astro
proc_lsst.multi INFO: found job 17001185 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:07 proc_lsst.multi:146 [INFO]  found job 17001186 in provider astro
proc_lsst.multi INFO: found job 17001186 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001186 PD
17001185 PD
17001184 PD
17001183 PD
17001182 PD
17001181 PD
17001180 PD
17001179 PD
17001178 PD
17001177 PD
17001176 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:07 proc_lsst.multi:146 [INFO]  found job 17001187 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001187 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001191 PD
17001190 PD
17001189 PD
17001188 PD
17001187 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:07 proc_lsst.multi:146 [INFO]  found job 17001188 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001188 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001191 PD
17001190 PD
17001189 PD
17001188 PD
17001187 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:08 proc_lsst.multi:146 [INFO]  found job 17001189 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001189 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001191 PD
17001190 PD
17001189 PD
17001188 PD
17001187 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:08 proc_lsst.multi:146 [INFO]  found job 17001190 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001190 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35fc7be801d17c7572222c4dcc43ee99 with result from task 5
parsl.dataflow.dflow DEBUG: Task 5 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/f4835ebb-2d42-4322-90e1-c40d75273ecc_isrForCrosstalkSources_898385_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/f4835ebb-2d42-4322-90e1-c40d75273ecc_isrForCrosstalkSources_898385_31.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001191 PD
17001190 PD
17001189 PD
17001188 PD
17001187 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:08 proc_lsst.multi:146 [INFO]  found job 17001191 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001191 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001191 PD
17001190 PD
17001189 PD
17001188 PD
17001187 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:08 proc_lsst.multi:146 [INFO]  found job 17001192 in provider escience_a40
proc_lsst.multi INFO: found job 17001192 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:09 proc_lsst.multi:146 [INFO]  found job 17001193 in provider escience_a40
proc_lsst.multi INFO: found job 17001193 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:09 proc_lsst.multi:146 [INFO]  found job 17001194 in provider escience_a40
proc_lsst.multi INFO: found job 17001194 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:09 proc_lsst.multi:146 [INFO]  found job 17001195 in provider escience_a40
proc_lsst.multi INFO: found job 17001195 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:09 proc_lsst.multi:146 [INFO]  found job 17001196 in provider escience_a40
proc_lsst.multi INFO: found job 17001196 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:10 proc_lsst.multi:146 [INFO]  found job 17001197 in provider escience_a40
proc_lsst.multi INFO: found job 17001197 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:10 proc_lsst.multi:146 [INFO]  found job 17001198 in provider escience_a40
proc_lsst.multi INFO: found job 17001198 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:10 proc_lsst.multi:146 [INFO]  found job 17001199 in provider escience_a40
proc_lsst.multi INFO: found job 17001199 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:11 proc_lsst.multi:146 [INFO]  found job 17001200 in provider escience_a40
proc_lsst.multi INFO: found job 17001200 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:11 proc_lsst.multi:146 [INFO]  found job 17001201 in provider escience_a40
proc_lsst.multi INFO: found job 17001201 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001211 PD
17001210 PD
17001209 PD
17001208 PD
17001207 PD
17001206 PD
17001205 PD
17001204 PD
17001203 PD
17001202 PD
17001201 PD
17001200 PD
17001199 PD
17001198 PD
17001197 PD
17001196 PD
17001195 PD
17001194 PD
17001193 PD
17001192 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:11 proc_lsst.multi:146 [INFO]  found job 17001202 in provider escience_a40
proc_lsst.multi INFO: found job 17001202 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:11 proc_lsst.multi:146 [INFO]  found job 17001203 in provider escience_a40
proc_lsst.multi INFO: found job 17001203 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:12 proc_lsst.multi:146 [INFO]  found job 17001204 in provider escience_a40
proc_lsst.multi INFO: found job 17001204 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:12 proc_lsst.multi:146 [INFO]  found job 17001205 in provider escience_a40
proc_lsst.multi INFO: found job 17001205 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:12 proc_lsst.multi:146 [INFO]  found job 17001206 in provider escience_a40
proc_lsst.multi INFO: found job 17001206 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:12 proc_lsst.multi:146 [INFO]  found job 17001207 in provider escience_a40
proc_lsst.multi INFO: found job 17001207 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:13 proc_lsst.multi:146 [INFO]  found job 17001208 in provider escience_a40
proc_lsst.multi INFO: found job 17001208 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:13 proc_lsst.multi:146 [INFO]  found job 17001209 in provider escience_a40
proc_lsst.multi INFO: found job 17001209 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:13 proc_lsst.multi:146 [INFO]  found job 17001210 in provider escience_a40
proc_lsst.multi INFO: found job 17001210 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:13 proc_lsst.multi:146 [INFO]  found job 17001211 in provider escience_a40
proc_lsst.multi INFO: found job 17001211 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:13 proc_lsst.multi:146 [INFO]  found job 17001212 in provider checkpoint
proc_lsst.multi INFO: found job 17001212 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:13 proc_lsst.multi:146 [INFO]  found job 17001213 in provider checkpoint
proc_lsst.multi INFO: found job 17001213 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:13 proc_lsst.multi:146 [INFO]  found job 17001214 in provider checkpoint
proc_lsst.multi INFO: found job 17001214 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:13 proc_lsst.multi:146 [INFO]  found job 17001215 in provider checkpoint
proc_lsst.multi INFO: found job 17001215 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:14 proc_lsst.multi:146 [INFO]  found job 17001216 in provider checkpoint
proc_lsst.multi INFO: found job 17001216 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:14 proc_lsst.multi:146 [INFO]  found job 17001217 in provider checkpoint
proc_lsst.multi INFO: found job 17001217 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:14 proc_lsst.multi:146 [INFO]  found job 17001218 in provider checkpoint
proc_lsst.multi INFO: found job 17001218 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:14 proc_lsst.multi:146 [INFO]  found job 17001219 in provider checkpoint
proc_lsst.multi INFO: found job 17001219 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:15 proc_lsst.multi:146 [INFO]  found job 17001220 in provider checkpoint
proc_lsst.multi INFO: found job 17001220 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:15 proc_lsst.multi:146 [INFO]  found job 17001221 in provider checkpoint
proc_lsst.multi INFO: found job 17001221 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:15 proc_lsst.multi:146 [INFO]  found job 17001222 in provider checkpoint
proc_lsst.multi INFO: found job 17001222 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:16 proc_lsst.multi:146 [INFO]  found job 17001223 in provider checkpoint
proc_lsst.multi INFO: found job 17001223 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52c7b28b0ab4f9b9f0a6818a83bd81bf with result from task 6
parsl.dataflow.dflow DEBUG: Task 6 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c16a69ca-a9bc-464b-b1b8-484f8ff75d9b_isrForCrosstalkSources_898280_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c16a69ca-a9bc-464b-b1b8-484f8ff75d9b_isrForCrosstalkSources_898280_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:16 proc_lsst.multi:146 [INFO]  found job 17001224 in provider checkpoint
proc_lsst.multi INFO: found job 17001224 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:16 proc_lsst.multi:146 [INFO]  found job 17001225 in provider checkpoint
proc_lsst.multi INFO: found job 17001225 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:16 proc_lsst.multi:146 [INFO]  found job 17001226 in provider checkpoint
proc_lsst.multi INFO: found job 17001226 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:17 proc_lsst.multi:146 [INFO]  found job 17001227 in provider checkpoint
proc_lsst.multi INFO: found job 17001227 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:17 proc_lsst.multi:146 [INFO]  found job 17001228 in provider checkpoint
proc_lsst.multi INFO: found job 17001228 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:17 proc_lsst.multi:146 [INFO]  found job 17001229 in provider checkpoint
proc_lsst.multi INFO: found job 17001229 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:18 proc_lsst.multi:146 [INFO]  found job 17001230 in provider checkpoint
proc_lsst.multi INFO: found job 17001230 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:18 proc_lsst.multi:146 [INFO]  found job 17001231 in provider checkpoint
proc_lsst.multi INFO: found job 17001231 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3717f5d8abe58acbb4e537f69297e374 with result from task 7
parsl.dataflow.dflow DEBUG: Task 7 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/f18d04ab-b8f6-4f57-ac6c-55ffcbcddd32_isrForCrosstalkSources_898282_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/f18d04ab-b8f6-4f57-ac6c-55ffcbcddd32_isrForCrosstalkSources_898282_38.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:18 proc_lsst.multi:146 [INFO]  found job 17001232 in provider checkpoint
proc_lsst.multi INFO: found job 17001232 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:19 proc_lsst.multi:146 [INFO]  found job 17001233 in provider checkpoint
proc_lsst.multi INFO: found job 17001233 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:19 proc_lsst.multi:146 [INFO]  found job 17001234 in provider checkpoint
proc_lsst.multi INFO: found job 17001234 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:19 proc_lsst.multi:146 [INFO]  found job 17001235 in provider checkpoint
proc_lsst.multi INFO: found job 17001235 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:19 proc_lsst.multi:146 [INFO]  found job 17001236 in provider checkpoint
proc_lsst.multi INFO: found job 17001236 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:20 proc_lsst.multi:146 [INFO]  found job 17001237 in provider checkpoint
proc_lsst.multi INFO: found job 17001237 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:20 proc_lsst.multi:146 [INFO]  found job 17001238 in provider checkpoint
proc_lsst.multi INFO: found job 17001238 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:20 proc_lsst.multi:146 [INFO]  found job 17001240 in provider checkpoint
proc_lsst.multi INFO: found job 17001240 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:21 proc_lsst.multi:146 [INFO]  found job 17001241 in provider checkpoint
proc_lsst.multi INFO: found job 17001241 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:21 proc_lsst.multi:146 [INFO]  found job 17001242 in provider checkpoint
proc_lsst.multi INFO: found job 17001242 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:21 proc_lsst.multi:146 [INFO]  found job 17001243 in provider checkpoint
proc_lsst.multi INFO: found job 17001243 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:22 proc_lsst.multi:146 [INFO]  found job 17001244 in provider checkpoint
proc_lsst.multi INFO: found job 17001244 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:22 proc_lsst.multi:146 [INFO]  found job 17001245 in provider checkpoint
proc_lsst.multi INFO: found job 17001245 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:22 proc_lsst.multi:146 [INFO]  found job 17001246 in provider checkpoint
proc_lsst.multi INFO: found job 17001246 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:23 proc_lsst.multi:146 [INFO]  found job 17001247 in provider checkpoint
proc_lsst.multi INFO: found job 17001247 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:23 proc_lsst.multi:146 [INFO]  found job 17001248 in provider checkpoint
proc_lsst.multi INFO: found job 17001248 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:23 proc_lsst.multi:146 [INFO]  found job 17001249 in provider checkpoint
proc_lsst.multi INFO: found job 17001249 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001212 PD
17001213 PD
17001214 PD
17001215 PD
17001216 PD
17001217 PD
17001218 PD
17001219 PD
17001220 PD
17001221 PD
17001222 PD
17001223 PD
17001224 PD
17001225 PD
17001226 PD
17001227 PD
17001228 PD
17001229 PD
17001230 PD
17001231 PD
17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 364
parsl.dataflow.strategy DEBUG: Executor multi has 364 active tasks, 1/92 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 364
parsl.dataflow.strategy DEBUG: Executor multi has 364 active tasks, 1/92 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d3c55233080a8f9be7a787ba09f7793 with result from task 8
parsl.dataflow.dflow DEBUG: Task 8 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/cae1c742-25b8-4a80-af3e-2e635e190bf3_isrForCrosstalkSources_898282_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/cae1c742-25b8-4a80-af3e-2e635e190bf3_isrForCrosstalkSources_898282_21.stderr
parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80a7f00a8fc879649bd4744a2b027556 with result from task 9
parsl.dataflow.dflow DEBUG: Task 9 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/a3da6af5-cd37-47ba-9ebb-98041c71aa30_isrForCrosstalkSources_898281_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/a3da6af5-cd37-47ba-9ebb-98041c71aa30_isrForCrosstalkSources_898281_59.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 362
parsl.dataflow.strategy DEBUG: Executor multi has 362 active tasks, 1/92 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 513ff602bc17cee8c13455908fe437e6 with result from task 10
parsl.dataflow.dflow DEBUG: Task 10 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/dee4bbee-c004-49df-bd85-9f67f273b108_isrForCrosstalkSources_898385_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/dee4bbee-c004-49df-bd85-9f67f273b108_isrForCrosstalkSources_898385_57.stderr
2024-03-11 23:01:33 proc_lsst.multi:146 [INFO]  found job 26021 in provider local
proc_lsst.multi INFO: found job 26021 in provider local
2024-03-11 23:01:33 proc_lsst.multi:146 [INFO]  found job 17001157 in provider astro
proc_lsst.multi INFO: found job 17001157 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:34 proc_lsst.multi:146 [INFO]  found job 17001158 in provider astro
proc_lsst.multi INFO: found job 17001158 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:34 proc_lsst.multi:146 [INFO]  found job 17001159 in provider astro
proc_lsst.multi INFO: found job 17001159 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:34 proc_lsst.multi:146 [INFO]  found job 17001160 in provider astro
proc_lsst.multi INFO: found job 17001160 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:35 proc_lsst.multi:146 [INFO]  found job 17001161 in provider astro
proc_lsst.multi INFO: found job 17001161 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:35 proc_lsst.multi:146 [INFO]  found job 17001162 in provider astro
proc_lsst.multi INFO: found job 17001162 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:35 proc_lsst.multi:146 [INFO]  found job 17001163 in provider astro
proc_lsst.multi INFO: found job 17001163 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:35 proc_lsst.multi:146 [INFO]  found job 17001164 in provider astro
proc_lsst.multi INFO: found job 17001164 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:36 proc_lsst.multi:146 [INFO]  found job 17001165 in provider astro
proc_lsst.multi INFO: found job 17001165 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:36 proc_lsst.multi:146 [INFO]  found job 17001166 in provider astro
proc_lsst.multi INFO: found job 17001166 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:36 proc_lsst.multi:146 [INFO]  found job 17001167 in provider astro
proc_lsst.multi INFO: found job 17001167 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:37 proc_lsst.multi:146 [INFO]  found job 17001168 in provider astro
proc_lsst.multi INFO: found job 17001168 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:37 proc_lsst.multi:146 [INFO]  found job 17001169 in provider astro
proc_lsst.multi INFO: found job 17001169 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:37 proc_lsst.multi:146 [INFO]  found job 17001170 in provider astro
proc_lsst.multi INFO: found job 17001170 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:38 proc_lsst.multi:146 [INFO]  found job 17001171 in provider astro
proc_lsst.multi INFO: found job 17001171 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 420e054faea92458bc3d5b9cc6aa6f11 with result from task 11
parsl.dataflow.dflow DEBUG: Task 11 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/346cc58c-809c-447b-8ef3-291001432c58_isrForCrosstalkSources_898280_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/346cc58c-809c-447b-8ef3-291001432c58_isrForCrosstalkSources_898280_9.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:38 proc_lsst.multi:146 [INFO]  found job 17001172 in provider astro
proc_lsst.multi INFO: found job 17001172 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:38 proc_lsst.multi:146 [INFO]  found job 17001173 in provider astro
proc_lsst.multi INFO: found job 17001173 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:38 proc_lsst.multi:146 [INFO]  found job 17001174 in provider astro
proc_lsst.multi INFO: found job 17001174 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:39 proc_lsst.multi:146 [INFO]  found job 17001175 in provider astro
proc_lsst.multi INFO: found job 17001175 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:39 proc_lsst.multi:146 [INFO]  found job 17001176 in provider astro
proc_lsst.multi INFO: found job 17001176 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:39 proc_lsst.multi:146 [INFO]  found job 17001177 in provider astro
proc_lsst.multi INFO: found job 17001177 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:40 proc_lsst.multi:146 [INFO]  found job 17001178 in provider astro
proc_lsst.multi INFO: found job 17001178 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:40 proc_lsst.multi:146 [INFO]  found job 17001179 in provider astro
proc_lsst.multi INFO: found job 17001179 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:40 proc_lsst.multi:146 [INFO]  found job 17001180 in provider astro
proc_lsst.multi INFO: found job 17001180 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:41 proc_lsst.multi:146 [INFO]  found job 17001181 in provider astro
proc_lsst.multi INFO: found job 17001181 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:41 proc_lsst.multi:146 [INFO]  found job 17001182 in provider astro
proc_lsst.multi INFO: found job 17001182 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:41 proc_lsst.multi:146 [INFO]  found job 17001183 in provider astro
proc_lsst.multi INFO: found job 17001183 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:42 proc_lsst.multi:146 [INFO]  found job 17001184 in provider astro
proc_lsst.multi INFO: found job 17001184 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:42 proc_lsst.multi:146 [INFO]  found job 17001185 in provider astro
proc_lsst.multi INFO: found job 17001185 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:42 proc_lsst.multi:146 [INFO]  found job 17001186 in provider astro
proc_lsst.multi INFO: found job 17001186 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:42 proc_lsst.multi:146 [INFO]  found job 17001187 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001187 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:42 proc_lsst.multi:146 [INFO]  found job 17001188 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001188 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:43 proc_lsst.multi:146 [INFO]  found job 17001189 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001189 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c69d94f8c403a3a03c2b56d024a9264 with result from task 12
parsl.dataflow.dflow DEBUG: Task 12 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/6ac9ddaf-f15c-4a50-b9a4-bcc11d6695a6_isrForCrosstalkSources_898386_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/6ac9ddaf-f15c-4a50-b9a4-bcc11d6695a6_isrForCrosstalkSources_898386_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:43 proc_lsst.multi:146 [INFO]  found job 17001190 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001190 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:43 proc_lsst.multi:146 [INFO]  found job 17001191 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001191 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:43 proc_lsst.multi:146 [INFO]  found job 17001192 in provider escience_a40
proc_lsst.multi INFO: found job 17001192 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:44 proc_lsst.multi:146 [INFO]  found job 17001193 in provider escience_a40
proc_lsst.multi INFO: found job 17001193 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:44 proc_lsst.multi:146 [INFO]  found job 17001194 in provider escience_a40
proc_lsst.multi INFO: found job 17001194 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:44 proc_lsst.multi:146 [INFO]  found job 17001195 in provider escience_a40
proc_lsst.multi INFO: found job 17001195 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:45 proc_lsst.multi:146 [INFO]  found job 17001196 in provider escience_a40
proc_lsst.multi INFO: found job 17001196 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:45 proc_lsst.multi:146 [INFO]  found job 17001197 in provider escience_a40
proc_lsst.multi INFO: found job 17001197 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:45 proc_lsst.multi:146 [INFO]  found job 17001198 in provider escience_a40
proc_lsst.multi INFO: found job 17001198 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:46 proc_lsst.multi:146 [INFO]  found job 17001199 in provider escience_a40
proc_lsst.multi INFO: found job 17001199 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:46 proc_lsst.multi:146 [INFO]  found job 17001200 in provider escience_a40
proc_lsst.multi INFO: found job 17001200 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:46 proc_lsst.multi:146 [INFO]  found job 17001201 in provider escience_a40
proc_lsst.multi INFO: found job 17001201 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:47 proc_lsst.multi:146 [INFO]  found job 17001202 in provider escience_a40
proc_lsst.multi INFO: found job 17001202 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:47 proc_lsst.multi:146 [INFO]  found job 17001203 in provider escience_a40
proc_lsst.multi INFO: found job 17001203 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:47 proc_lsst.multi:146 [INFO]  found job 17001204 in provider escience_a40
proc_lsst.multi INFO: found job 17001204 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d3ae44d160f0fce202da2e3e9d8917e7 with result from task 13
parsl.dataflow.dflow DEBUG: Task 13 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2ff55ecb-b7b2-495a-9b4d-576033e50d67_isrForCrosstalkSources_898282_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2ff55ecb-b7b2-495a-9b4d-576033e50d67_isrForCrosstalkSources_898282_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:47 proc_lsst.multi:146 [INFO]  found job 17001205 in provider escience_a40
proc_lsst.multi INFO: found job 17001205 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:48 proc_lsst.multi:146 [INFO]  found job 17001206 in provider escience_a40
proc_lsst.multi INFO: found job 17001206 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:48 proc_lsst.multi:146 [INFO]  found job 17001207 in provider escience_a40
proc_lsst.multi INFO: found job 17001207 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:48 proc_lsst.multi:146 [INFO]  found job 17001208 in provider escience_a40
proc_lsst.multi INFO: found job 17001208 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:48 proc_lsst.multi:146 [INFO]  found job 17001209 in provider escience_a40
proc_lsst.multi INFO: found job 17001209 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:48 proc_lsst.multi:146 [INFO]  found job 17001210 in provider escience_a40
proc_lsst.multi INFO: found job 17001210 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:49 proc_lsst.multi:146 [INFO]  found job 17001211 in provider escience_a40
proc_lsst.multi INFO: found job 17001211 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:01:49 proc_lsst.multi:146 [INFO]  found job 17001212 in provider checkpoint
proc_lsst.multi INFO: found job 17001212 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:49 proc_lsst.multi:146 [INFO]  found job 17001213 in provider checkpoint
proc_lsst.multi INFO: found job 17001213 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:49 proc_lsst.multi:146 [INFO]  found job 17001214 in provider checkpoint
proc_lsst.multi INFO: found job 17001214 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:50 proc_lsst.multi:146 [INFO]  found job 17001215 in provider checkpoint
proc_lsst.multi INFO: found job 17001215 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:50 proc_lsst.multi:146 [INFO]  found job 17001216 in provider checkpoint
proc_lsst.multi INFO: found job 17001216 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:50 proc_lsst.multi:146 [INFO]  found job 17001217 in provider checkpoint
proc_lsst.multi INFO: found job 17001217 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:51 proc_lsst.multi:146 [INFO]  found job 17001218 in provider checkpoint
proc_lsst.multi INFO: found job 17001218 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:51 proc_lsst.multi:146 [INFO]  found job 17001219 in provider checkpoint
proc_lsst.multi INFO: found job 17001219 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:51 proc_lsst.multi:146 [INFO]  found job 17001220 in provider checkpoint
proc_lsst.multi INFO: found job 17001220 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:52 proc_lsst.multi:146 [INFO]  found job 17001221 in provider checkpoint
proc_lsst.multi INFO: found job 17001221 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:52 proc_lsst.multi:146 [INFO]  found job 17001222 in provider checkpoint
proc_lsst.multi INFO: found job 17001222 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:52 proc_lsst.multi:146 [INFO]  found job 17001223 in provider checkpoint
proc_lsst.multi INFO: found job 17001223 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:52 proc_lsst.multi:146 [INFO]  found job 17001224 in provider checkpoint
proc_lsst.multi INFO: found job 17001224 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:52 proc_lsst.multi:146 [INFO]  found job 17001225 in provider checkpoint
proc_lsst.multi INFO: found job 17001225 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:53 proc_lsst.multi:146 [INFO]  found job 17001226 in provider checkpoint
proc_lsst.multi INFO: found job 17001226 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e0032e36910d48b9f87b793edd9c490a with result from task 14
parsl.dataflow.dflow DEBUG: Task 14 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7753cdbd-048d-4206-af67-96435b5587f9_isrForCrosstalkSources_898385_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7753cdbd-048d-4206-af67-96435b5587f9_isrForCrosstalkSources_898385_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:53 proc_lsst.multi:146 [INFO]  found job 17001227 in provider checkpoint
proc_lsst.multi INFO: found job 17001227 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:53 proc_lsst.multi:146 [INFO]  found job 17001228 in provider checkpoint
proc_lsst.multi INFO: found job 17001228 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:53 proc_lsst.multi:146 [INFO]  found job 17001229 in provider checkpoint
proc_lsst.multi INFO: found job 17001229 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:54 proc_lsst.multi:146 [INFO]  found job 17001230 in provider checkpoint
proc_lsst.multi INFO: found job 17001230 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:54 proc_lsst.multi:146 [INFO]  found job 17001231 in provider checkpoint
proc_lsst.multi INFO: found job 17001231 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:54 proc_lsst.multi:146 [INFO]  found job 17001232 in provider checkpoint
proc_lsst.multi INFO: found job 17001232 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:55 proc_lsst.multi:146 [INFO]  found job 17001233 in provider checkpoint
proc_lsst.multi INFO: found job 17001233 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:55 proc_lsst.multi:146 [INFO]  found job 17001234 in provider checkpoint
proc_lsst.multi INFO: found job 17001234 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:55 proc_lsst.multi:146 [INFO]  found job 17001235 in provider checkpoint
proc_lsst.multi INFO: found job 17001235 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:56 proc_lsst.multi:146 [INFO]  found job 17001236 in provider checkpoint
proc_lsst.multi INFO: found job 17001236 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:56 proc_lsst.multi:146 [INFO]  found job 17001237 in provider checkpoint
proc_lsst.multi INFO: found job 17001237 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:56 proc_lsst.multi:146 [INFO]  found job 17001238 in provider checkpoint
proc_lsst.multi INFO: found job 17001238 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:56 proc_lsst.multi:146 [INFO]  found job 17001240 in provider checkpoint
proc_lsst.multi INFO: found job 17001240 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:57 proc_lsst.multi:146 [INFO]  found job 17001241 in provider checkpoint
proc_lsst.multi INFO: found job 17001241 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b5f5c5455e1ee9cb7e81278f05611a1 with result from task 15
parsl.dataflow.dflow DEBUG: Task 15 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/dcc1328a-8406-471f-a410-092227f93719_isrForCrosstalkSources_898281_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/dcc1328a-8406-471f-a410-092227f93719_isrForCrosstalkSources_898281_13.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:57 proc_lsst.multi:146 [INFO]  found job 17001242 in provider checkpoint
proc_lsst.multi INFO: found job 17001242 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:57 proc_lsst.multi:146 [INFO]  found job 17001243 in provider checkpoint
proc_lsst.multi INFO: found job 17001243 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:57 proc_lsst.multi:146 [INFO]  found job 17001244 in provider checkpoint
proc_lsst.multi INFO: found job 17001244 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:58 proc_lsst.multi:146 [INFO]  found job 17001245 in provider checkpoint
proc_lsst.multi INFO: found job 17001245 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:58 proc_lsst.multi:146 [INFO]  found job 17001246 in provider checkpoint
proc_lsst.multi INFO: found job 17001246 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:58 proc_lsst.multi:146 [INFO]  found job 17001247 in provider checkpoint
proc_lsst.multi INFO: found job 17001247 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:59 proc_lsst.multi:146 [INFO]  found job 17001248 in provider checkpoint
proc_lsst.multi INFO: found job 17001248 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:01:59 proc_lsst.multi:146 [INFO]  found job 17001249 in provider checkpoint
proc_lsst.multi INFO: found job 17001249 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001249 PD
17001248 PD
17001247 PD
17001246 PD
17001245 PD
17001244 PD
17001243 PD
17001242 PD
17001241 PD
17001240 PD
17001238 PD
17001237 PD
17001236 PD
17001235 PD
17001234 PD
17001233 PD
17001232 PD
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 356
parsl.dataflow.strategy DEBUG: Executor multi has 356 active tasks, 21/72 running/pending blocks, and 162 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 356
parsl.dataflow.strategy DEBUG: Executor multi has 356 active tasks, 21/72 running/pending blocks, and 162 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 885a563d12b8c4f1fde90c52c4b7f8a7 with result from task 16
parsl.dataflow.dflow DEBUG: Task 16 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/17eec91b-d77f-4afd-8eeb-0d6181b36860_isrForCrosstalkSources_898280_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/17eec91b-d77f-4afd-8eeb-0d6181b36860_isrForCrosstalkSources_898280_13.stderr
2024-03-11 23:02:04 proc_lsst.multi:146 [INFO]  found job 26021 in provider local
proc_lsst.multi INFO: found job 26021 in provider local
2024-03-11 23:02:04 proc_lsst.multi:146 [INFO]  found job 17001157 in provider astro
proc_lsst.multi INFO: found job 17001157 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:05 proc_lsst.multi:146 [INFO]  found job 17001158 in provider astro
proc_lsst.multi INFO: found job 17001158 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:05 proc_lsst.multi:146 [INFO]  found job 17001159 in provider astro
proc_lsst.multi INFO: found job 17001159 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:05 proc_lsst.multi:146 [INFO]  found job 17001160 in provider astro
proc_lsst.multi INFO: found job 17001160 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:06 proc_lsst.multi:146 [INFO]  found job 17001161 in provider astro
proc_lsst.multi INFO: found job 17001161 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:06 proc_lsst.multi:146 [INFO]  found job 17001162 in provider astro
proc_lsst.multi INFO: found job 17001162 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 74c83a57394281c52e550eb0bdd2fcb3 with result from task 41
parsl.dataflow.dflow DEBUG: Task 41 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/abf88f43-a134-4870-b03f-8bf50c98055c_isrForCrosstalkSources_898386_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/abf88f43-a134-4870-b03f-8bf50c98055c_isrForCrosstalkSources_898386_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:06 proc_lsst.multi:146 [INFO]  found job 17001163 in provider astro
proc_lsst.multi INFO: found job 17001163 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:06 proc_lsst.multi:146 [INFO]  found job 17001164 in provider astro
proc_lsst.multi INFO: found job 17001164 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:07 proc_lsst.multi:146 [INFO]  found job 17001165 in provider astro
proc_lsst.multi INFO: found job 17001165 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:07 proc_lsst.multi:146 [INFO]  found job 17001166 in provider astro
proc_lsst.multi INFO: found job 17001166 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:07 proc_lsst.multi:146 [INFO]  found job 17001167 in provider astro
proc_lsst.multi INFO: found job 17001167 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:08 proc_lsst.multi:146 [INFO]  found job 17001168 in provider astro
proc_lsst.multi INFO: found job 17001168 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:08 proc_lsst.multi:146 [INFO]  found job 17001169 in provider astro
proc_lsst.multi INFO: found job 17001169 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:08 proc_lsst.multi:146 [INFO]  found job 17001170 in provider astro
proc_lsst.multi INFO: found job 17001170 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:08 proc_lsst.multi:146 [INFO]  found job 17001171 in provider astro
proc_lsst.multi INFO: found job 17001171 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:09 proc_lsst.multi:146 [INFO]  found job 17001172 in provider astro
proc_lsst.multi INFO: found job 17001172 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:09 proc_lsst.multi:146 [INFO]  found job 17001173 in provider astro
proc_lsst.multi INFO: found job 17001173 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:09 proc_lsst.multi:146 [INFO]  found job 17001174 in provider astro
proc_lsst.multi INFO: found job 17001174 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:10 proc_lsst.multi:146 [INFO]  found job 17001175 in provider astro
proc_lsst.multi INFO: found job 17001175 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:10 proc_lsst.multi:146 [INFO]  found job 17001176 in provider astro
proc_lsst.multi INFO: found job 17001176 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:10 proc_lsst.multi:146 [INFO]  found job 17001177 in provider astro
proc_lsst.multi INFO: found job 17001177 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:10 proc_lsst.multi:146 [INFO]  found job 17001178 in provider astro
proc_lsst.multi INFO: found job 17001178 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:11 proc_lsst.multi:146 [INFO]  found job 17001179 in provider astro
proc_lsst.multi INFO: found job 17001179 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:11 proc_lsst.multi:146 [INFO]  found job 17001180 in provider astro
proc_lsst.multi INFO: found job 17001180 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:11 proc_lsst.multi:146 [INFO]  found job 17001181 in provider astro
proc_lsst.multi INFO: found job 17001181 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:12 proc_lsst.multi:146 [INFO]  found job 17001182 in provider astro
proc_lsst.multi INFO: found job 17001182 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52ad14e025dcab58efefbd1492a57dc3 with result from task 178
parsl.dataflow.dflow DEBUG: Task 178 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f1266bdc-a44c-4dbe-a88a-23be5ab7759f_isrForCrosstalkSources_898386_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f1266bdc-a44c-4dbe-a88a-23be5ab7759f_isrForCrosstalkSources_898386_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:12 proc_lsst.multi:146 [INFO]  found job 17001183 in provider astro
proc_lsst.multi INFO: found job 17001183 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:12 proc_lsst.multi:146 [INFO]  found job 17001184 in provider astro
proc_lsst.multi INFO: found job 17001184 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:12 proc_lsst.multi:146 [INFO]  found job 17001185 in provider astro
proc_lsst.multi INFO: found job 17001185 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:13 proc_lsst.multi:146 [INFO]  found job 17001186 in provider astro
proc_lsst.multi INFO: found job 17001186 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:13 proc_lsst.multi:146 [INFO]  found job 17001187 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001187 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:13 proc_lsst.multi:146 [INFO]  found job 17001188 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001188 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:14 proc_lsst.multi:146 [INFO]  found job 17001189 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001189 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:14 proc_lsst.multi:146 [INFO]  found job 17001190 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001190 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:14 proc_lsst.multi:146 [INFO]  found job 17001191 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001191 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:14 proc_lsst.multi:146 [INFO]  found job 17001192 in provider escience_a40
proc_lsst.multi INFO: found job 17001192 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:15 proc_lsst.multi:146 [INFO]  found job 17001193 in provider escience_a40
proc_lsst.multi INFO: found job 17001193 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a23cff98c736c2d6abd17f3469998d23 with result from task 179
parsl.dataflow.dflow DEBUG: Task 179 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/7eac29c7-e6c0-4ee9-9bff-410c1ac65464_isrForCrosstalkSources_898386_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/7eac29c7-e6c0-4ee9-9bff-410c1ac65464_isrForCrosstalkSources_898386_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:15 proc_lsst.multi:146 [INFO]  found job 17001194 in provider escience_a40
proc_lsst.multi INFO: found job 17001194 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:15 proc_lsst.multi:146 [INFO]  found job 17001195 in provider escience_a40
proc_lsst.multi INFO: found job 17001195 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:16 proc_lsst.multi:146 [INFO]  found job 17001196 in provider escience_a40
proc_lsst.multi INFO: found job 17001196 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:16 proc_lsst.multi:146 [INFO]  found job 17001197 in provider escience_a40
proc_lsst.multi INFO: found job 17001197 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:16 proc_lsst.multi:146 [INFO]  found job 17001198 in provider escience_a40
proc_lsst.multi INFO: found job 17001198 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:17 proc_lsst.multi:146 [INFO]  found job 17001199 in provider escience_a40
proc_lsst.multi INFO: found job 17001199 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:17 proc_lsst.multi:146 [INFO]  found job 17001200 in provider escience_a40
proc_lsst.multi INFO: found job 17001200 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:19 proc_lsst.multi:146 [INFO]  found job 17001201 in provider escience_a40
proc_lsst.multi INFO: found job 17001201 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:19 proc_lsst.multi:146 [INFO]  found job 17001202 in provider escience_a40
proc_lsst.multi INFO: found job 17001202 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:20 proc_lsst.multi:146 [INFO]  found job 17001203 in provider escience_a40
proc_lsst.multi INFO: found job 17001203 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:20 proc_lsst.multi:146 [INFO]  found job 17001204 in provider escience_a40
proc_lsst.multi INFO: found job 17001204 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:20 proc_lsst.multi:146 [INFO]  found job 17001205 in provider escience_a40
proc_lsst.multi INFO: found job 17001205 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:20 proc_lsst.multi:146 [INFO]  found job 17001206 in provider escience_a40
proc_lsst.multi INFO: found job 17001206 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:21 proc_lsst.multi:146 [INFO]  found job 17001207 in provider escience_a40
proc_lsst.multi INFO: found job 17001207 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:21 proc_lsst.multi:146 [INFO]  found job 17001208 in provider escience_a40
proc_lsst.multi INFO: found job 17001208 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:21 proc_lsst.multi:146 [INFO]  found job 17001209 in provider escience_a40
proc_lsst.multi INFO: found job 17001209 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:21 proc_lsst.multi:146 [INFO]  found job 17001210 in provider escience_a40
proc_lsst.multi INFO: found job 17001210 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:21 proc_lsst.multi:146 [INFO]  found job 17001211 in provider escience_a40
proc_lsst.multi INFO: found job 17001211 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:22 proc_lsst.multi:146 [INFO]  found job 17001212 in provider checkpoint
proc_lsst.multi INFO: found job 17001212 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:22 proc_lsst.multi:146 [INFO]  found job 17001213 in provider checkpoint
proc_lsst.multi INFO: found job 17001213 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:22 proc_lsst.multi:146 [INFO]  found job 17001214 in provider checkpoint
proc_lsst.multi INFO: found job 17001214 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:22 proc_lsst.multi:146 [INFO]  found job 17001215 in provider checkpoint
proc_lsst.multi INFO: found job 17001215 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20165d74ee47cba94dbf0b9932a46e07 with result from task 180
parsl.dataflow.dflow DEBUG: Task 180 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7dbff09f-527d-4bae-b6ef-a0e0cb62c63f_isrForCrosstalkSources_898385_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7dbff09f-527d-4bae-b6ef-a0e0cb62c63f_isrForCrosstalkSources_898385_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:22 proc_lsst.multi:146 [INFO]  found job 17001216 in provider checkpoint
proc_lsst.multi INFO: found job 17001216 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:23 proc_lsst.multi:146 [INFO]  found job 17001217 in provider checkpoint
proc_lsst.multi INFO: found job 17001217 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:23 proc_lsst.multi:146 [INFO]  found job 17001218 in provider checkpoint
proc_lsst.multi INFO: found job 17001218 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:23 proc_lsst.multi:146 [INFO]  found job 17001219 in provider checkpoint
proc_lsst.multi INFO: found job 17001219 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:23 proc_lsst.multi:146 [INFO]  found job 17001220 in provider checkpoint
proc_lsst.multi INFO: found job 17001220 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:23 proc_lsst.multi:146 [INFO]  found job 17001221 in provider checkpoint
proc_lsst.multi INFO: found job 17001221 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:24 proc_lsst.multi:146 [INFO]  found job 17001222 in provider checkpoint
proc_lsst.multi INFO: found job 17001222 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:24 proc_lsst.multi:146 [INFO]  found job 17001223 in provider checkpoint
proc_lsst.multi INFO: found job 17001223 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:24 proc_lsst.multi:146 [INFO]  found job 17001224 in provider checkpoint
proc_lsst.multi INFO: found job 17001224 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:24 proc_lsst.multi:146 [INFO]  found job 17001225 in provider checkpoint
proc_lsst.multi INFO: found job 17001225 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:24 proc_lsst.multi:146 [INFO]  found job 17001226 in provider checkpoint
proc_lsst.multi INFO: found job 17001226 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:25 proc_lsst.multi:146 [INFO]  found job 17001227 in provider checkpoint
proc_lsst.multi INFO: found job 17001227 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:25 proc_lsst.multi:146 [INFO]  found job 17001228 in provider checkpoint
proc_lsst.multi INFO: found job 17001228 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc5d0457a7005fbe6e220dc389b337ad with result from task 181
parsl.dataflow.dflow DEBUG: Task 181 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/85f5afe2-7f69-4df8-b154-523d4378d3f5_isrForCrosstalkSources_898282_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/85f5afe2-7f69-4df8-b154-523d4378d3f5_isrForCrosstalkSources_898282_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:25 proc_lsst.multi:146 [INFO]  found job 17001229 in provider checkpoint
proc_lsst.multi INFO: found job 17001229 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:26 proc_lsst.multi:146 [INFO]  found job 17001230 in provider checkpoint
proc_lsst.multi INFO: found job 17001230 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:26 proc_lsst.multi:146 [INFO]  found job 17001231 in provider checkpoint
proc_lsst.multi INFO: found job 17001231 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:26 proc_lsst.multi:146 [INFO]  found job 17001232 in provider checkpoint
proc_lsst.multi INFO: found job 17001232 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2ebd7dc9b28f41968eb8e9e3b6ee73e6 with result from task 68
parsl.dataflow.dflow DEBUG: Task 68 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7baffd0c-6318-4a86-bf4c-03f939394d66_isrForCrosstalkSources_898281_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7baffd0c-6318-4a86-bf4c-03f939394d66_isrForCrosstalkSources_898281_9.stderr
parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88ea11a911edf92ea645580cafcacf5e with result from task 73
parsl.dataflow.dflow DEBUG: Task 73 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1d034d2f-e6da-492c-bf1d-6faf3522e076_isrForCrosstalkSources_898282_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1d034d2f-e6da-492c-bf1d-6faf3522e076_isrForCrosstalkSources_898282_27.stderr
parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8051e81374da6f4adcbdb25a9bbab40 with result from task 69
parsl.dataflow.dflow DEBUG: Task 69 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3605e328-a18a-4c69-acf4-38b9f86ef674_isrForCrosstalkSources_898387_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3605e328-a18a-4c69-acf4-38b9f86ef674_isrForCrosstalkSources_898387_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:27 proc_lsst.multi:146 [INFO]  found job 17001233 in provider checkpoint
proc_lsst.multi INFO: found job 17001233 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:27 proc_lsst.multi:146 [INFO]  found job 17001234 in provider checkpoint
proc_lsst.multi INFO: found job 17001234 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 029b95783a9863ec95eefdee130ec605 with result from task 70
parsl.dataflow.dflow DEBUG: Task 70 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/035d1fb3-de5b-410d-a039-e3d6577a6965_isrForCrosstalkSources_898386_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/035d1fb3-de5b-410d-a039-e3d6577a6965_isrForCrosstalkSources_898386_1.stderr
parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23ebac5b4012236bffbd75e5f7bd8686 with result from task 72
parsl.dataflow.dflow DEBUG: Task 72 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/d276d934-433b-4b0b-9a67-66158cb2e14a_isrForCrosstalkSources_898385_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/d276d934-433b-4b0b-9a67-66158cb2e14a_isrForCrosstalkSources_898385_10.stderr
parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8c2fc3dc6175cb2a80712633cc93fc0 with result from task 71
parsl.dataflow.dflow DEBUG: Task 71 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7cd41d53-6dd0-4b2e-adee-8e0a05c2e843_isrForCrosstalkSources_898385_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7cd41d53-6dd0-4b2e-adee-8e0a05c2e843_isrForCrosstalkSources_898385_29.stderr
parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b89e66aa51523b4a6faddf5322505880 with result from task 67
parsl.dataflow.dflow DEBUG: Task 67 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1d4c7c27-928d-453d-953e-cf6f90f00fb8_isrForCrosstalkSources_898282_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1d4c7c27-928d-453d-953e-cf6f90f00fb8_isrForCrosstalkSources_898282_31.stderr
parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c99b9d6f5825761198bac21a482044b with result from task 66
parsl.dataflow.dflow DEBUG: Task 66 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/b222d146-0c59-4c48-a286-ae23f4b95738_isrForCrosstalkSources_898386_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/b222d146-0c59-4c48-a286-ae23f4b95738_isrForCrosstalkSources_898386_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:27 proc_lsst.multi:146 [INFO]  found job 17001235 in provider checkpoint
proc_lsst.multi INFO: found job 17001235 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry de3ec7bda6305a18308a01ee01bb2254 with result from task 42
parsl.dataflow.dflow DEBUG: Task 42 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/826c11e3-7b71-4bce-9741-eae5eed04a7a_isrForCrosstalkSources_898386_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/826c11e3-7b71-4bce-9741-eae5eed04a7a_isrForCrosstalkSources_898386_4.stderr
parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b62488298adb43db8b17fbff1ae1a27 with result from task 47
parsl.dataflow.dflow DEBUG: Task 47 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c79f167f-8e27-41d0-a80e-58de9a33826a_isrForCrosstalkSources_898282_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c79f167f-8e27-41d0-a80e-58de9a33826a_isrForCrosstalkSources_898282_41.stderr
parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad97f3914a37e5b02af130822ba11a7e with result from task 49
parsl.dataflow.dflow DEBUG: Task 49 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/27bc8482-4453-4fbf-99b1-4f3a673b4b28_isrForCrosstalkSources_898386_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/27bc8482-4453-4fbf-99b1-4f3a673b4b28_isrForCrosstalkSources_898386_32.stderr
parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91745207ed642774a743cd90f56b71d1 with result from task 45
parsl.dataflow.dflow DEBUG: Task 45 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/759c3fc2-e728-4d6a-adff-d78ffabc95a3_isrForCrosstalkSources_898387_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/759c3fc2-e728-4d6a-adff-d78ffabc95a3_isrForCrosstalkSources_898387_21.stderr
parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f39462653ace5ab3806b8479dbda6085 with result from task 48
parsl.dataflow.dflow DEBUG: Task 48 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/ec3c5e95-bf39-4440-b2fd-3f3c11fb36ea_isrForCrosstalkSources_898282_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/ec3c5e95-bf39-4440-b2fd-3f3c11fb36ea_isrForCrosstalkSources_898282_23.stderr
parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18a2b4d4c48b2d0a9643b9828bf24975 with result from task 46
parsl.dataflow.dflow DEBUG: Task 46 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/d5d0b886-8bd8-4376-a37f-5f7a7ce9f534_isrForCrosstalkSources_898280_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/d5d0b886-8bd8-4376-a37f-5f7a7ce9f534_isrForCrosstalkSources_898280_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:28 proc_lsst.multi:146 [INFO]  found job 17001236 in provider checkpoint
proc_lsst.multi INFO: found job 17001236 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry adab158eadf1f429f9d20c17f4b45a83 with result from task 43
parsl.dataflow.dflow DEBUG: Task 43 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/dfe186ea-5d2b-4b7f-aeb5-8b2d93546cdc_isrForCrosstalkSources_898280_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/dfe186ea-5d2b-4b7f-aeb5-8b2d93546cdc_isrForCrosstalkSources_898280_16.stderr
parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7acdac1e3505a316c1da629bd992d66e with result from task 44
parsl.dataflow.dflow DEBUG: Task 44 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/0da73840-5d80-4d93-9e41-6b4cc2432474_isrForCrosstalkSources_898280_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/0da73840-5d80-4d93-9e41-6b4cc2432474_isrForCrosstalkSources_898280_29.stderr
parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f66613753d89e86d88dbbc739363e151 with result from task 26
parsl.dataflow.dflow DEBUG: Task 26 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b9f828d8-a2d0-40f9-8610-b9d329bf30a9_isrForCrosstalkSources_898282_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b9f828d8-a2d0-40f9-8610-b9d329bf30a9_isrForCrosstalkSources_898282_17.stderr
parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6f7cae6b73ffe97519f9b27d206174c with result from task 29
parsl.dataflow.dflow DEBUG: Task 29 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bc40e429-4e5c-460e-9d70-35065e086810_isrForCrosstalkSources_898282_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bc40e429-4e5c-460e-9d70-35065e086810_isrForCrosstalkSources_898282_29.stderr
parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 618cd96555ea1e68f17ff6c1566fe106 with result from task 27
parsl.dataflow.dflow DEBUG: Task 27 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/321865f4-8612-464c-a481-8f132f4c263a_isrForCrosstalkSources_898281_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/321865f4-8612-464c-a481-8f132f4c263a_isrForCrosstalkSources_898281_43.stderr
parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dcc410a4a01ae0a5137e4ca02474af01 with result from task 28
parsl.dataflow.dflow DEBUG: Task 28 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d88b873d-cb41-4024-8ee3-2f5fe70f4756_isrForCrosstalkSources_898387_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d88b873d-cb41-4024-8ee3-2f5fe70f4756_isrForCrosstalkSources_898387_7.stderr
parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba5315b8ce456764b60508be2d48bfc8 with result from task 31
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:28 proc_lsst.multi:146 [INFO]  found job 17001237 in provider checkpoint
proc_lsst.multi INFO: found job 17001237 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow DEBUG: Task 31 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/039b7c8f-1cb2-43f2-a38e-67a6ad248127_isrForCrosstalkSources_898387_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/039b7c8f-1cb2-43f2-a38e-67a6ad248127_isrForCrosstalkSources_898387_52.stderr
parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f58fb1a4d2691383d4fd81852bdce24b with result from task 30
parsl.dataflow.dflow DEBUG: Task 30 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/6917180e-6c4a-4126-bed4-fc5f56134766_isrForCrosstalkSources_898282_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/6917180e-6c4a-4126-bed4-fc5f56134766_isrForCrosstalkSources_898282_22.stderr
parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c4b109187beca62538e07bf6968b7ab with result from task 32
parsl.dataflow.dflow DEBUG: Task 32 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/635a7426-b3ad-4662-882f-a61e98e19d94_isrForCrosstalkSources_898387_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/635a7426-b3ad-4662-882f-a61e98e19d94_isrForCrosstalkSources_898387_4.stderr
parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb974b2aac315105cb406022bf0916c8 with result from task 25
parsl.dataflow.dflow DEBUG: Task 25 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/6caeb0fc-eedc-4e34-96b0-9a6aea420ee0_isrForCrosstalkSources_898387_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/6caeb0fc-eedc-4e34-96b0-9a6aea420ee0_isrForCrosstalkSources_898387_58.stderr
parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed298d258ec787a7215ba929fce85e8a with result from task 18
parsl.dataflow.dflow DEBUG: Task 18 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/eb30a0c0-81ee-40ad-a344-19d1c6e832b2_isrForCrosstalkSources_898282_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/eb30a0c0-81ee-40ad-a344-19d1c6e832b2_isrForCrosstalkSources_898282_51.stderr
parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5387d8e16a6fccd8f2900e215541411e with result from task 24
parsl.dataflow.dflow DEBUG: Task 24 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/79b75c9d-077c-468c-8338-7205a6c9c41a_isrForCrosstalkSources_898386_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/79b75c9d-077c-468c-8338-7205a6c9c41a_isrForCrosstalkSources_898386_35.stderr
parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eaba84f095013a01360f6f57ea021b53 with result from task 21
parsl.dataflow.dflow DEBUG: Task 21 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/d391c3ef-c633-4039-b849-f78ab141ee93_isrForCrosstalkSources_898281_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/d391c3ef-c633-4039-b849-f78ab141ee93_isrForCrosstalkSources_898281_16.stderr
parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb4bf52421a254f1d7e5ab2272ab0a48 with result from task 20
parsl.dataflow.dflow DEBUG: Task 20 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/bead777e-069e-4dd5-96e0-35923c5bd1b3_isrForCrosstalkSources_898385_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/bead777e-069e-4dd5-96e0-35923c5bd1b3_isrForCrosstalkSources_898385_24.stderr
parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87ba32e8034a25bd2d7307e410f9e355 with result from task 17
parsl.dataflow.dflow DEBUG: Task 17 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/573546e3-2299-40f5-aef8-894eb4975890_isrForCrosstalkSources_898385_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/573546e3-2299-40f5-aef8-894eb4975890_isrForCrosstalkSources_898385_3.stderr
parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 431fbec67b1e77161ff8aa712991f01a with result from task 22
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:28 proc_lsst.multi:146 [INFO]  found job 17001238 in provider checkpoint
proc_lsst.multi INFO: found job 17001238 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow DEBUG: Task 22 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/6e950a10-9a4f-420f-9ff3-044ea88f8804_isrForCrosstalkSources_898386_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/6e950a10-9a4f-420f-9ff3-044ea88f8804_isrForCrosstalkSources_898386_29.stderr
parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a4ae9fbcf3cfc6296244e4fe3eef11ef with result from task 19
parsl.dataflow.dflow DEBUG: Task 19 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b86a6793-05f9-48b9-aebe-0be1b346d28c_isrForCrosstalkSources_898282_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b86a6793-05f9-48b9-aebe-0be1b346d28c_isrForCrosstalkSources_898282_24.stderr
parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d1195ad3eb3a2fcedcee37148a66dd04 with result from task 23
parsl.dataflow.dflow DEBUG: Task 23 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4cfe8fd0-2867-4f4c-8179-738f586e8e26_isrForCrosstalkSources_898282_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4cfe8fd0-2867-4f4c-8179-738f586e8e26_isrForCrosstalkSources_898282_61.stderr
parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 07893403f8feb2c5bbc249db7de7c19d with result from task 56
parsl.dataflow.dflow DEBUG: Task 56 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c97dbb63-c753-4d6d-971d-f0f69b1bbddc_isrForCrosstalkSources_898280_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c97dbb63-c753-4d6d-971d-f0f69b1bbddc_isrForCrosstalkSources_898280_14.stderr
parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6ba26ba3d612f7c1d4a0c862e306776 with result from task 50
parsl.dataflow.dflow DEBUG: Task 50 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/08feae9c-1183-4ffe-ac12-6b5e366c193b_isrForCrosstalkSources_898386_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/08feae9c-1183-4ffe-ac12-6b5e366c193b_isrForCrosstalkSources_898386_26.stderr
parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 90a31db2e5cfef3240148e93f11bba03 with result from task 55
parsl.dataflow.dflow DEBUG: Task 55 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/01be10c7-f039-4fb5-b263-ac9df914ebc0_isrForCrosstalkSources_898282_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/01be10c7-f039-4fb5-b263-ac9df914ebc0_isrForCrosstalkSources_898282_60.stderr
parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca7d8015cf2e51224a9b0e6d11c1e6df with result from task 53
parsl.dataflow.dflow DEBUG: Task 53 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3ebae64f-a5bc-4441-a7f8-e2dd028bff49_isrForCrosstalkSources_898387_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3ebae64f-a5bc-4441-a7f8-e2dd028bff49_isrForCrosstalkSources_898387_42.stderr
parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c42832a753ac8eb60baf04cb8a47c9c3 with result from task 57
parsl.dataflow.dflow DEBUG: Task 57 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3e6f47e5-6772-41cb-80d1-851f43edfb14_isrForCrosstalkSources_898385_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3e6f47e5-6772-41cb-80d1-851f43edfb14_isrForCrosstalkSources_898385_9.stderr
parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c92b1e815588d425db1b6338cb749fe with result from task 54
parsl.dataflow.dflow DEBUG: Task 54 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/bfd1bf58-2a19-473e-9555-068a03263f4a_isrForCrosstalkSources_898387_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/bfd1bf58-2a19-473e-9555-068a03263f4a_isrForCrosstalkSources_898387_14.stderr
parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7546d6afece851de1a9a849df089a099 with result from task 51
parsl.dataflow.dflow DEBUG: Task 51 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/9ca8a2b7-4ecb-4039-b5e8-82827328df46_isrForCrosstalkSources_898282_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/9ca8a2b7-4ecb-4039-b5e8-82827328df46_isrForCrosstalkSources_898282_40.stderr
parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 158e677845e19b5baf5ed6f02fd8a5ec with result from task 77
parsl.dataflow.dflow DEBUG: Task 77 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7a3a47e3-12b3-4ded-9000-20c0712f6263_isrForCrosstalkSources_898281_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7a3a47e3-12b3-4ded-9000-20c0712f6263_isrForCrosstalkSources_898281_33.stderr
parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6517f16bba02343fb6698177e06df193 with result from task 74
parsl.dataflow.dflow DEBUG: Task 74 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/bd1b8ae4-f06f-469b-a43d-b9b75603a4db_isrForCrosstalkSources_898387_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/bd1b8ae4-f06f-469b-a43d-b9b75603a4db_isrForCrosstalkSources_898387_13.stderr
parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d1d79d1b5d60817053827a3f4908427 with result from task 79
parsl.dataflow.dflow DEBUG: Task 79 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e0a76c0b-fe8f-4eac-a8a8-8c657903f457_isrForCrosstalkSources_898385_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e0a76c0b-fe8f-4eac-a8a8-8c657903f457_isrForCrosstalkSources_898385_34.stderr
parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e81e065b1add20377bdc3c8ed660bf7 with result from task 80
parsl.dataflow.dflow DEBUG: Task 80 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/fb675716-bc83-44df-83fb-7968e7ae12bb_isrForCrosstalkSources_898387_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/fb675716-bc83-44df-83fb-7968e7ae12bb_isrForCrosstalkSources_898387_3.stderr
parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79fa6cdaf0939371949eaeaa4b0126b9 with result from task 87
parsl.dataflow.dflow DEBUG: Task 87 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3eef727a-0277-4e19-9752-824b77fc713f_isrForCrosstalkSources_898387_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3eef727a-0277-4e19-9752-824b77fc713f_isrForCrosstalkSources_898387_56.stderr
parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1880fd1649c8e00817c90cde21562b0b with result from task 89
parsl.dataflow.dflow DEBUG: Task 89 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6b2163ec-620a-44f6-8052-b7173e55124f_isrForCrosstalkSources_898385_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6b2163ec-620a-44f6-8052-b7173e55124f_isrForCrosstalkSources_898385_8.stderr
parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e1cc4ae0f2d57a1021ea6d88855a70f3 with result from task 86
parsl.dataflow.dflow DEBUG: Task 86 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/2be23c92-824a-4045-aeee-61272cc2bfca_isrForCrosstalkSources_898280_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/2be23c92-824a-4045-aeee-61272cc2bfca_isrForCrosstalkSources_898280_40.stderr
parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6600472cc4538e0234e26df543213e9f with result from task 83
parsl.dataflow.dflow DEBUG: Task 83 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/388c99c9-6b57-42b4-a78d-be022dd70c4c_isrForCrosstalkSources_898281_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/388c99c9-6b57-42b4-a78d-be022dd70c4c_isrForCrosstalkSources_898281_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:29 proc_lsst.multi:146 [INFO]  found job 17001240 in provider checkpoint
proc_lsst.multi INFO: found job 17001240 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2bbe4913834eee1dd1701ea686ee4424 with result from task 134
parsl.dataflow.dflow DEBUG: Task 134 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/45b6ec5b-bdff-4572-9f95-b1f8e39f3746_isrForCrosstalkSources_898387_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/45b6ec5b-bdff-4572-9f95-b1f8e39f3746_isrForCrosstalkSources_898387_32.stderr
parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ce6ea97e0b4a225e962f251a44917de with result from task 141
parsl.dataflow.dflow DEBUG: Task 141 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7363ec24-9a59-4e77-bc84-0a0257ed0f35_isrForCrosstalkSources_898385_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7363ec24-9a59-4e77-bc84-0a0257ed0f35_isrForCrosstalkSources_898385_14.stderr
parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 727cd2f56b3ce3f56f8920aa19f54c54 with result from task 145
parsl.dataflow.dflow DEBUG: Task 145 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2ebe8bdf-f7fa-4a24-9664-4cf278e591eb_isrForCrosstalkSources_898281_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2ebe8bdf-f7fa-4a24-9664-4cf278e591eb_isrForCrosstalkSources_898281_29.stderr
parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 08276e167965e02656d4387041ec7aff with result from task 143
parsl.dataflow.dflow DEBUG: Task 143 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2df9bfa7-36e5-430e-80e2-f7745c7aa294_isrForCrosstalkSources_898281_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2df9bfa7-36e5-430e-80e2-f7745c7aa294_isrForCrosstalkSources_898281_15.stderr
parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b961b6c2d534c79a235903f941fc7f39 with result from task 139
parsl.dataflow.dflow DEBUG: Task 139 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7c4fa937-6298-4033-943e-26a0d765e94c_isrForCrosstalkSources_898387_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7c4fa937-6298-4033-943e-26a0d765e94c_isrForCrosstalkSources_898387_40.stderr
parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b487121cdc717e50ed086360f2ef3ff6 with result from task 52
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:29 proc_lsst.multi:146 [INFO]  found job 17001241 in provider checkpoint
proc_lsst.multi INFO: found job 17001241 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow DEBUG: Task 52 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4a00fc89-2d0c-4934-8233-484d2e4e47a6_isrForCrosstalkSources_898281_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4a00fc89-2d0c-4934-8233-484d2e4e47a6_isrForCrosstalkSources_898281_31.stderr
parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c6afe8d8143783e0086ef21ab09cdfa with result from task 75
parsl.dataflow.dflow DEBUG: Task 75 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d86c89e4-b375-49a2-b838-526aa51de49e_isrForCrosstalkSources_898282_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d86c89e4-b375-49a2-b838-526aa51de49e_isrForCrosstalkSources_898282_42.stderr
parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5117ee1456b4ba0329dd9246ce3e4978 with result from task 78
parsl.dataflow.dflow DEBUG: Task 78 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/8e6c74a2-7be1-4e0d-a5d6-457a7c639163_isrForCrosstalkSources_898280_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/8e6c74a2-7be1-4e0d-a5d6-457a7c639163_isrForCrosstalkSources_898280_54.stderr
parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 40ef0f4666bee99f8aec0783b39a44b0 with result from task 81
parsl.dataflow.dflow DEBUG: Task 81 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/0e902274-5344-458a-a19d-88be97803d9b_isrForCrosstalkSources_898280_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/0e902274-5344-458a-a19d-88be97803d9b_isrForCrosstalkSources_898280_57.stderr
parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a23a36d398ba4fd4b92bf2a304d0dbe8 with result from task 76
parsl.dataflow.dflow DEBUG: Task 76 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e0554f2e-3eff-4502-89df-18eb0935c9a1_isrForCrosstalkSources_898281_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e0554f2e-3eff-4502-89df-18eb0935c9a1_isrForCrosstalkSources_898281_28.stderr
parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2ed14c1ef7e1f4a49220a2bdb17d580b with result from task 92
parsl.dataflow.dflow DEBUG: Task 92 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e651db0b-e51f-4e8a-a236-c0a75be2bd9a_isrForCrosstalkSources_898385_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e651db0b-e51f-4e8a-a236-c0a75be2bd9a_isrForCrosstalkSources_898385_4.stderr
parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 139fbd3862fe536d711d389ff6812a03 with result from task 95
parsl.dataflow.dflow DEBUG: Task 95 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0b9fd929-150a-49c4-bf69-4fa09e172c13_isrForCrosstalkSources_898386_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0b9fd929-150a-49c4-bf69-4fa09e172c13_isrForCrosstalkSources_898386_3.stderr
parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 992032f1a5296d9b21eb849afc11a406 with result from task 94
parsl.dataflow.dflow DEBUG: Task 94 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/337bb5b3-b0a1-4186-b474-0db9cb2465f4_isrForCrosstalkSources_898280_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/337bb5b3-b0a1-4186-b474-0db9cb2465f4_isrForCrosstalkSources_898280_50.stderr
parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ec652b67163e34074342e802efaa100 with result from task 90
parsl.dataflow.dflow DEBUG: Task 90 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/fb77a77c-5d96-44ed-9db3-6b030d582a77_isrForCrosstalkSources_898386_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/fb77a77c-5d96-44ed-9db3-6b030d582a77_isrForCrosstalkSources_898386_22.stderr
parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 178eca08806e2172676896c0cfc5e521 with result from task 97
parsl.dataflow.dflow DEBUG: Task 97 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e935ec2d-3239-44c9-ac60-069079cf2bbb_isrForCrosstalkSources_898280_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e935ec2d-3239-44c9-ac60-069079cf2bbb_isrForCrosstalkSources_898280_43.stderr
parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 454fbc1a8d4c96f618e0e3b2e518eca7 with result from task 93
parsl.dataflow.dflow DEBUG: Task 93 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/04e34fe9-99b1-4178-970d-47400b6cb66c_isrForCrosstalkSources_898385_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/04e34fe9-99b1-4178-970d-47400b6cb66c_isrForCrosstalkSources_898385_28.stderr
parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46bfc325a8642502f1e9e1b5a1f7960f with result from task 96
parsl.dataflow.dflow DEBUG: Task 96 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/4a1fd99d-cc97-4243-ada9-ca367939300c_isrForCrosstalkSources_898387_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/4a1fd99d-cc97-4243-ada9-ca367939300c_isrForCrosstalkSources_898387_12.stderr
parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a150a765e43b7e0c6143becc98592490 with result from task 91
parsl.dataflow.dflow DEBUG: Task 91 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/b82afdd6-28f7-41f2-a652-508bb35f3262_isrForCrosstalkSources_898385_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/b82afdd6-28f7-41f2-a652-508bb35f3262_isrForCrosstalkSources_898385_33.stderr
parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 508f948e3212ec9e3c2f3b4dd2602bae with result from task 130
parsl.dataflow.dflow DEBUG: Task 130 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f5a33d6b-32fe-410f-8d84-584a184e9220_isrForCrosstalkSources_898386_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f5a33d6b-32fe-410f-8d84-584a184e9220_isrForCrosstalkSources_898386_33.stderr
parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5a630e700fa51208409ea3d1f71ddfc0 with result from task 151
parsl.dataflow.dflow DEBUG: Task 151 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/dcc4f17a-a929-4b77-8cf1-3d0a7198c754_isrForCrosstalkSources_898385_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/dcc4f17a-a929-4b77-8cf1-3d0a7198c754_isrForCrosstalkSources_898385_35.stderr
parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 77f5718780bec33120caabd46514c62b with result from task 153
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:29 proc_lsst.multi:146 [INFO]  found job 17001242 in provider checkpoint
proc_lsst.multi INFO: found job 17001242 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow DEBUG: Task 153 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/893ac501-c1fb-4933-b2cb-f4a5ee430188_isrForCrosstalkSources_898281_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/893ac501-c1fb-4933-b2cb-f4a5ee430188_isrForCrosstalkSources_898281_26.stderr
parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 21f1e089ea742186b8a4f609394b7e85 with result from task 138
parsl.dataflow.dflow DEBUG: Task 138 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/623f79be-83be-412d-9028-9239d10a717f_isrForCrosstalkSources_898386_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/623f79be-83be-412d-9028-9239d10a717f_isrForCrosstalkSources_898386_48.stderr
parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f704ea47ddb38e4e424f02a7af9e0b7f with result from task 82
parsl.dataflow.dflow DEBUG: Task 82 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/f094ca45-3c8f-4336-8e85-9c9b9ad6c809_isrForCrosstalkSources_898280_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/f094ca45-3c8f-4336-8e85-9c9b9ad6c809_isrForCrosstalkSources_898280_55.stderr
parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0cfd825556a087234ebb1d207224bd7 with result from task 136
parsl.dataflow.dflow DEBUG: Task 136 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/95daf612-8458-4f07-adfd-15a9cea0da9a_isrForCrosstalkSources_898281_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/95daf612-8458-4f07-adfd-15a9cea0da9a_isrForCrosstalkSources_898281_51.stderr
parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1b4d74d2f42036696fa99ea8d0613326 with result from task 88
parsl.dataflow.dflow DEBUG: Task 88 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/2bbb5ec3-7871-4f36-ac2f-82bce59ab85c_isrForCrosstalkSources_898387_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/2bbb5ec3-7871-4f36-ac2f-82bce59ab85c_isrForCrosstalkSources_898387_53.stderr
parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 804d810e542c513696c11d619b045c37 with result from task 132
parsl.dataflow.dflow DEBUG: Task 132 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f5544d53-eeac-4d1a-be75-f7243c0c91d0_isrForCrosstalkSources_898386_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f5544d53-eeac-4d1a-be75-f7243c0c91d0_isrForCrosstalkSources_898386_14.stderr
parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fba5a1028a1a629d1ca36ae63f0c1706 with result from task 152
parsl.dataflow.dflow DEBUG: Task 152 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/eaf86225-fdba-43b2-8b11-2cead9adf579_isrForCrosstalkSources_898387_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/eaf86225-fdba-43b2-8b11-2cead9adf579_isrForCrosstalkSources_898387_30.stderr
parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a63890c7b5d4ecad48993e12ec6040b6 with result from task 146
parsl.dataflow.dflow DEBUG: Task 146 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/28347066-a84e-49ee-bb2b-72198923c3b2_isrForCrosstalkSources_898280_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/28347066-a84e-49ee-bb2b-72198923c3b2_isrForCrosstalkSources_898280_7.stderr
parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fba0c96e304921360edd7785e3c19831 with result from task 103
parsl.dataflow.dflow DEBUG: Task 103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7e1e93f5-2130-405b-869a-14797fb47763_isrForCrosstalkSources_898281_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7e1e93f5-2130-405b-869a-14797fb47763_isrForCrosstalkSources_898281_14.stderr
parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b08d859e803673aa97b433fa98dfb1e with result from task 102
parsl.dataflow.dflow DEBUG: Task 102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/edaa34e5-083d-4208-88d2-728fef6f79dd_isrForCrosstalkSources_898386_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/edaa34e5-083d-4208-88d2-728fef6f79dd_isrForCrosstalkSources_898386_11.stderr
parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2dc072d97b344741198a7907d4523277 with result from task 98
parsl.dataflow.dflow DEBUG: Task 98 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/4f3fc0b9-f8d4-4389-8d38-8057ff6eade3_isrForCrosstalkSources_898387_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/4f3fc0b9-f8d4-4389-8d38-8057ff6eade3_isrForCrosstalkSources_898387_8.stderr
parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93e2289944be28ba6af2562e12f64c93 with result from task 112
parsl.dataflow.dflow DEBUG: Task 112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/74ab14bd-3354-4b28-b8bb-0b8ab86a2923_isrForCrosstalkSources_898281_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/74ab14bd-3354-4b28-b8bb-0b8ab86a2923_isrForCrosstalkSources_898281_20.stderr
parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fd3236decffe114f058d1c9a83420f1a with result from task 107
parsl.dataflow.dflow DEBUG: Task 107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5d669546-ba4d-463f-b32a-bdcf48b34860_isrForCrosstalkSources_898385_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5d669546-ba4d-463f-b32a-bdcf48b34860_isrForCrosstalkSources_898385_13.stderr
parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11b3bf88cf2994f0c94bc6ab12fa53ed with result from task 104
parsl.dataflow.dflow DEBUG: Task 104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/13503b45-f338-49bb-b518-c1883eeaf47f_isrForCrosstalkSources_898387_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/13503b45-f338-49bb-b518-c1883eeaf47f_isrForCrosstalkSources_898387_10.stderr
parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry addef5db32c507019249e235e287fa07 with result from task 105
parsl.dataflow.dflow DEBUG: Task 105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/d2f86bba-e154-4264-88f9-9c55c44bd70e_isrForCrosstalkSources_898386_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/d2f86bba-e154-4264-88f9-9c55c44bd70e_isrForCrosstalkSources_898386_10.stderr
parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3041cb7991f40d668978555f41f3d58 with result from task 85
parsl.dataflow.dflow DEBUG: Task 85 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/754d1e88-0451-43dc-a9d3-6f9d09545b00_isrForCrosstalkSources_898385_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/754d1e88-0451-43dc-a9d3-6f9d09545b00_isrForCrosstalkSources_898385_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:30 proc_lsst.multi:146 [INFO]  found job 17001243 in provider checkpoint
proc_lsst.multi INFO: found job 17001243 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55cd500f18fdc62c3b741d727da31637 with result from task 150
parsl.dataflow.dflow DEBUG: Task 150 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/9f05c5c3-decb-401c-884f-0d55c22afa35_isrForCrosstalkSources_898281_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/9f05c5c3-decb-401c-884f-0d55c22afa35_isrForCrosstalkSources_898281_56.stderr
parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry deac855a19991494628ee1082bf86e24 with result from task 84
parsl.dataflow.dflow DEBUG: Task 84 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/91e5e6c0-fb82-4690-8598-e503b7933d59_isrForCrosstalkSources_898385_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/91e5e6c0-fb82-4690-8598-e503b7933d59_isrForCrosstalkSources_898385_42.stderr
parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 066cb37aa10f3829aa68c394153e9f22 with result from task 110
parsl.dataflow.dflow DEBUG: Task 110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/559cdd8b-168b-424f-a57c-733f119f8fea_isrForCrosstalkSources_898280_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/559cdd8b-168b-424f-a57c-733f119f8fea_isrForCrosstalkSources_898280_21.stderr
parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca75def372d49171b50e8a9af9872a73 with result from task 109
parsl.dataflow.dflow DEBUG: Task 109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/9825ae25-8802-4976-b6c6-d8c357848e9f_isrForCrosstalkSources_898386_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/9825ae25-8802-4976-b6c6-d8c357848e9f_isrForCrosstalkSources_898386_18.stderr
parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5ffa39e85260f2e3dfef2f881f22d20 with result from task 133
parsl.dataflow.dflow DEBUG: Task 133 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/56467881-8553-4be9-9662-fcef57f6cb85_isrForCrosstalkSources_898387_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/56467881-8553-4be9-9662-fcef57f6cb85_isrForCrosstalkSources_898387_50.stderr
parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e975c1eee28d3db3644152d246a4d5b with result from task 137
parsl.dataflow.dflow DEBUG: Task 137 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ab9d14cc-0c27-4c91-a4da-0135758cfe47_isrForCrosstalkSources_898387_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ab9d14cc-0c27-4c91-a4da-0135758cfe47_isrForCrosstalkSources_898387_60.stderr
parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1dbfe3a254967cee5d142192d169ff1f with result from task 147
parsl.dataflow.dflow DEBUG: Task 147 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2cc4d09b-3f08-4f49-941d-4a71450eb8dd_isrForCrosstalkSources_898282_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2cc4d09b-3f08-4f49-941d-4a71450eb8dd_isrForCrosstalkSources_898282_12.stderr
parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 92b92d57dfe9688ed3b1f5b686fae6d4 with result from task 148
parsl.dataflow.dflow DEBUG: Task 148 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/31206e2b-a2b7-4114-ac85-1f6d903a7284_isrForCrosstalkSources_898282_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/31206e2b-a2b7-4114-ac85-1f6d903a7284_isrForCrosstalkSources_898282_37.stderr
parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c01827d37e5f65fffb4dff2a3b8b3519 with result from task 149
parsl.dataflow.dflow DEBUG: Task 149 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c300cccb-912f-4964-a2cc-b01116971fe9_isrForCrosstalkSources_898282_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c300cccb-912f-4964-a2cc-b01116971fe9_isrForCrosstalkSources_898282_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:30 proc_lsst.multi:146 [INFO]  found job 17001244 in provider checkpoint
proc_lsst.multi INFO: found job 17001244 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a6f604fb4f3430acf5a2e367b7600c6 with result from task 142
parsl.dataflow.dflow DEBUG: Task 142 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e1180f62-bce8-4c0e-b97e-c4a114357a64_isrForCrosstalkSources_898280_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e1180f62-bce8-4c0e-b97e-c4a114357a64_isrForCrosstalkSources_898280_24.stderr
parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 77b6f24758d7afbc678e52113a6d8f5e with result from task 61
parsl.dataflow.dflow DEBUG: Task 61 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/f8c033b4-f0f3-45ba-8c86-99d72ca31380_isrForCrosstalkSources_898387_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/f8c033b4-f0f3-45ba-8c86-99d72ca31380_isrForCrosstalkSources_898387_54.stderr
parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 04634525748a86d4bb659d7d639a1954 with result from task 62
parsl.dataflow.dflow DEBUG: Task 62 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/24c51d62-064e-49ab-b50c-e2563f91942e_isrForCrosstalkSources_898387_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/24c51d62-064e-49ab-b50c-e2563f91942e_isrForCrosstalkSources_898387_26.stderr
parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52679cf0b2dd089a100c18882d8912cc with result from task 58
parsl.dataflow.dflow DEBUG: Task 58 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ff49fdfd-3173-4eea-8c58-e6b3e3e461eb_isrForCrosstalkSources_898387_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ff49fdfd-3173-4eea-8c58-e6b3e3e461eb_isrForCrosstalkSources_898387_46.stderr
parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0414502efd09092a6898f86e45eb4d69 with result from task 65
parsl.dataflow.dflow DEBUG: Task 65 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/a2c4b4bd-abcf-47c4-847a-dccbd275fb58_isrForCrosstalkSources_898280_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/a2c4b4bd-abcf-47c4-847a-dccbd275fb58_isrForCrosstalkSources_898280_48.stderr
parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 15f0b396ebc3628671470927ed18ff80 with result from task 64
parsl.dataflow.dflow DEBUG: Task 64 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/77ba637f-fef6-4452-87fb-26e18549ceef_isrForCrosstalkSources_898280_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/77ba637f-fef6-4452-87fb-26e18549ceef_isrForCrosstalkSources_898280_20.stderr
parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d39f9430738af9e04bb863f868196cb with result from task 60
parsl.dataflow.dflow DEBUG: Task 60 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/a0e225e9-81a4-491c-be26-1832651b0081_isrForCrosstalkSources_898282_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/a0e225e9-81a4-491c-be26-1832651b0081_isrForCrosstalkSources_898282_10.stderr
parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a64aac11bac6d44d1520d7b53b759bf1 with result from task 135
parsl.dataflow.dflow DEBUG: Task 135 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/91be2ad1-db28-4c64-924f-f151b582d7a9_isrForCrosstalkSources_898385_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/91be2ad1-db28-4c64-924f-f151b582d7a9_isrForCrosstalkSources_898385_50.stderr
parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aaeb3802b3a44e256c6626505503b3cf with result from task 144
parsl.dataflow.dflow DEBUG: Task 144 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/f0bb9889-4558-4728-b09b-562ef77f6994_isrForCrosstalkSources_898280_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/f0bb9889-4558-4728-b09b-562ef77f6994_isrForCrosstalkSources_898280_23.stderr
parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 57a114670ed709fe7b2f0fae934b25d4 with result from task 106
parsl.dataflow.dflow DEBUG: Task 106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/87d1e268-4ace-4936-bd9c-0b0c2968c098_isrForCrosstalkSources_898385_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/87d1e268-4ace-4936-bd9c-0b0c2968c098_isrForCrosstalkSources_898385_11.stderr
parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b2d57b9f55a3fcb7075ebfd28043728 with result from task 113
parsl.dataflow.dflow DEBUG: Task 113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/46db20f2-d2d2-453e-9e87-92a064193082_isrForCrosstalkSources_898386_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/46db20f2-d2d2-453e-9e87-92a064193082_isrForCrosstalkSources_898386_31.stderr
parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 547db9253d8884640238321a4deb0737 with result from task 59
parsl.dataflow.dflow DEBUG: Task 59 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/514e4fdc-5a5f-488b-8dbb-bfff6a9db519_isrForCrosstalkSources_898281_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/514e4fdc-5a5f-488b-8dbb-bfff6a9db519_isrForCrosstalkSources_898281_49.stderr
parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14df738597db02a2b7a1569ce332103d with result from task 111
parsl.dataflow.dflow DEBUG: Task 111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/38ffc2a8-afe2-4e6e-b032-d673eeb4c6fc_isrForCrosstalkSources_898280_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/38ffc2a8-afe2-4e6e-b032-d673eeb4c6fc_isrForCrosstalkSources_898280_31.stderr
parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f58971163f5395fdbf6ae2dbac0f7ce with result from task 63
parsl.dataflow.dflow DEBUG: Task 63 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/8be8d598-abbb-463d-8dff-6c7fefda6790_isrForCrosstalkSources_898386_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/8be8d598-abbb-463d-8dff-6c7fefda6790_isrForCrosstalkSources_898386_60.stderr
parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88e6e1d4e87f81d5957daf4e73cbf0f1 with result from task 108
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:30 proc_lsst.multi:146 [INFO]  found job 17001245 in provider checkpoint
proc_lsst.multi INFO: found job 17001245 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow DEBUG: Task 108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/56b7d0a8-d906-4941-a6ff-fb366aeb9055_isrForCrosstalkSources_898387_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/56b7d0a8-d906-4941-a6ff-fb366aeb9055_isrForCrosstalkSources_898387_2.stderr
parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd282fa8573fb0ea1b02d58c48b6d230 with result from task 100
parsl.dataflow.dflow DEBUG: Task 100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/4e63639d-6651-4a4a-ac9f-bff48ec7cfe6_isrForCrosstalkSources_898386_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/4e63639d-6651-4a4a-ac9f-bff48ec7cfe6_isrForCrosstalkSources_898386_30.stderr
parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14ea10625b0a9408bc77f4b0f56fdeeb with result from task 131
parsl.dataflow.dflow DEBUG: Task 131 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/7fe59330-e716-44db-8d07-4193d49cebe1_isrForCrosstalkSources_898280_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/7fe59330-e716-44db-8d07-4193d49cebe1_isrForCrosstalkSources_898280_56.stderr
parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00896763f662c3926b479cc9b4e5a858 with result from task 125
parsl.dataflow.dflow DEBUG: Task 125 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b05d69a7-b7fb-4d30-a88d-970c0af1d11d_isrForCrosstalkSources_898282_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b05d69a7-b7fb-4d30-a88d-970c0af1d11d_isrForCrosstalkSources_898282_47.stderr
parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd1a4cffff823606c864479921762c5 with result from task 126
parsl.dataflow.dflow DEBUG: Task 126 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2d298252-6b82-4eb2-9f21-9696a1f4752f_isrForCrosstalkSources_898282_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2d298252-6b82-4eb2-9f21-9696a1f4752f_isrForCrosstalkSources_898282_58.stderr
parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 746dc3a759211ed54e2cd42e5a019520 with result from task 122
parsl.dataflow.dflow DEBUG: Task 122 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b435c2ac-4290-4bf5-8e76-0a45f881eadb_isrForCrosstalkSources_898282_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b435c2ac-4290-4bf5-8e76-0a45f881eadb_isrForCrosstalkSources_898282_16.stderr
parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c962aba4980ee47a89e9076560408a7c with result from task 128
parsl.dataflow.dflow DEBUG: Task 128 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/109496d3-91f7-4400-a0ae-37fef025e2cc_isrForCrosstalkSources_898280_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/109496d3-91f7-4400-a0ae-37fef025e2cc_isrForCrosstalkSources_898280_5.stderr
parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry beef2f4b92772d63164372699efc61ac with result from task 129
parsl.dataflow.dflow DEBUG: Task 129 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/a2910115-4816-49fc-99be-328dbd5cbacb_isrForCrosstalkSources_898280_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/a2910115-4816-49fc-99be-328dbd5cbacb_isrForCrosstalkSources_898280_59.stderr
parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f015d75f50eba1dcce05eb46fca89c54 with result from task 124
parsl.dataflow.dflow DEBUG: Task 124 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/dc82db18-28d2-4482-a06c-f9380187a673_isrForCrosstalkSources_898386_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/dc82db18-28d2-4482-a06c-f9380187a673_isrForCrosstalkSources_898386_45.stderr
parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 12f2cf69c0dce8e3d83b0c6da05a0c91 with result from task 127
parsl.dataflow.dflow DEBUG: Task 127 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/48ccb5a2-1065-4ea7-bd1e-dc18dda8cd83_isrForCrosstalkSources_898387_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/48ccb5a2-1065-4ea7-bd1e-dc18dda8cd83_isrForCrosstalkSources_898387_48.stderr
parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37b6020dc871ea6e580e7a25b51f20a5 with result from task 123
parsl.dataflow.dflow DEBUG: Task 123 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5d2bbb3c-c7d1-47e7-886e-50087a7e336a_isrForCrosstalkSources_898385_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5d2bbb3c-c7d1-47e7-886e-50087a7e336a_isrForCrosstalkSources_898385_61.stderr
parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1bdaf16dbac79324094dbf4419a5bea7 with result from task 37
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:31 proc_lsst.multi:146 [INFO]  found job 17001246 in provider checkpoint
proc_lsst.multi INFO: found job 17001246 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow DEBUG: Task 37 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/15bdc7f3-9bc4-443b-bae4-afdf02ee76fe_isrForCrosstalkSources_898281_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/15bdc7f3-9bc4-443b-bae4-afdf02ee76fe_isrForCrosstalkSources_898281_3.stderr
parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aebfbd9cd1d96f6c0099067496c5ea61 with result from task 33
parsl.dataflow.dflow DEBUG: Task 33 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/8caf94f5-495c-4b77-aaaa-6f735d7d7ce2_isrForCrosstalkSources_898282_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/8caf94f5-495c-4b77-aaaa-6f735d7d7ce2_isrForCrosstalkSources_898282_36.stderr
parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 17b9893eea0e427ef7400a1c09060559 with result from task 34
parsl.dataflow.dflow DEBUG: Task 34 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/402c66fc-737b-46e0-914e-a26375b729af_isrForCrosstalkSources_898386_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/402c66fc-737b-46e0-914e-a26375b729af_isrForCrosstalkSources_898386_46.stderr
parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3092d181b97d4faadb9921f7355fedc with result from task 35
parsl.dataflow.dflow DEBUG: Task 35 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/f3dd7e49-7940-4eeb-8cba-b244236d7937_isrForCrosstalkSources_898387_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/f3dd7e49-7940-4eeb-8cba-b244236d7937_isrForCrosstalkSources_898387_9.stderr
parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46c13619215e51d75e926ac67187d5d6 with result from task 38
parsl.dataflow.dflow DEBUG: Task 38 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bd138c97-a6c3-4744-bc05-c8814b914e94_isrForCrosstalkSources_898281_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bd138c97-a6c3-4744-bc05-c8814b914e94_isrForCrosstalkSources_898281_60.stderr
parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef7a45a96f7383ffb5525e18b549ae55 with result from task 39
parsl.dataflow.dflow DEBUG: Task 39 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/60a5a6d5-c04b-49d8-8129-a4ab43e0a21d_isrForCrosstalkSources_898282_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/60a5a6d5-c04b-49d8-8129-a4ab43e0a21d_isrForCrosstalkSources_898282_48.stderr
parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d3f9959c15372d9cc0c8de8f9dd8e0d with result from task 40
parsl.dataflow.dflow DEBUG: Task 40 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/5b444700-cb54-40e5-b3f3-52df420641b9_isrForCrosstalkSources_898386_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/5b444700-cb54-40e5-b3f3-52df420641b9_isrForCrosstalkSources_898386_7.stderr
parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f9632ca4dae868d4f01bace07d1c105 with result from task 36
parsl.dataflow.dflow DEBUG: Task 36 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/fc75db1d-355f-4032-a303-e530543d4707_isrForCrosstalkSources_898280_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/fc75db1d-355f-4032-a303-e530543d4707_isrForCrosstalkSources_898280_38.stderr
parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72c0b0a97ee429e74d34d574e3316063 with result from task 140
parsl.dataflow.dflow DEBUG: Task 140 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/aaab69eb-7387-4823-b94d-457e92b6f9c3_isrForCrosstalkSources_898280_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/aaab69eb-7387-4823-b94d-457e92b6f9c3_isrForCrosstalkSources_898280_1.stderr
parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81e528c91fb052577ac2fb42967c9b04 with result from task 101
parsl.dataflow.dflow DEBUG: Task 101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/affd6abb-27f7-4a88-9f80-fc5c5cb58dea_isrForCrosstalkSources_898385_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/affd6abb-27f7-4a88-9f80-fc5c5cb58dea_isrForCrosstalkSources_898385_59.stderr
parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 398b9bbdbb866fee980f2abf20867b36 with result from task 168
parsl.dataflow.dflow DEBUG: Task 168 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6774ce4f-ef02-4960-a55c-314fb976987f_isrForCrosstalkSources_898385_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6774ce4f-ef02-4960-a55c-314fb976987f_isrForCrosstalkSources_898385_53.stderr
parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 73a892624061891803c9d89fee918a7b with result from task 169
parsl.dataflow.dflow DEBUG: Task 169 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3fde7376-9e80-4012-9ee3-a39c2d5b4aff_isrForCrosstalkSources_898385_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3fde7376-9e80-4012-9ee3-a39c2d5b4aff_isrForCrosstalkSources_898385_52.stderr
parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5664e94fd38bc5d0cc5118b08a7897c9 with result from task 165
parsl.dataflow.dflow DEBUG: Task 165 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2e548571-7f9e-4de3-8dc8-9eb7139b646a_isrForCrosstalkSources_898386_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2e548571-7f9e-4de3-8dc8-9eb7139b646a_isrForCrosstalkSources_898386_16.stderr
parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa242fcf0027821dd6db22c6a742695 with result from task 163
parsl.dataflow.dflow DEBUG: Task 163 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/5ea83cd1-52df-411d-9df3-707c0895a288_isrForCrosstalkSources_898387_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/5ea83cd1-52df-411d-9df3-707c0895a288_isrForCrosstalkSources_898387_49.stderr
parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f984f422cf9ef9932339f492949c1ce5 with result from task 166
parsl.dataflow.dflow DEBUG: Task 166 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/c02b9ba2-40ee-4acc-854f-88f517d25d02_isrForCrosstalkSources_898281_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/c02b9ba2-40ee-4acc-854f-88f517d25d02_isrForCrosstalkSources_898281_10.stderr
parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e578bd0571df2d50e020af5dc2bb9024 with result from task 99
parsl.dataflow.dflow DEBUG: Task 99 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/38fecc07-a335-4cb0-a785-108cffe74cb7_isrForCrosstalkSources_898280_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/38fecc07-a335-4cb0-a785-108cffe74cb7_isrForCrosstalkSources_898280_53.stderr
parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a07125a4c7426d650555891f3b89c36f with result from task 162
parsl.dataflow.dflow DEBUG: Task 162 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/9cf61144-d391-4ea5-99a5-d0ca96505e26_isrForCrosstalkSources_898281_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/9cf61144-d391-4ea5-99a5-d0ca96505e26_isrForCrosstalkSources_898281_50.stderr
parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c54014058bf8f70f551e538a7e4a55d with result from task 172
parsl.dataflow.dflow DEBUG: Task 172 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5a416810-3e06-423c-95e9-4dfba26671c1_isrForCrosstalkSources_898385_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5a416810-3e06-423c-95e9-4dfba26671c1_isrForCrosstalkSources_898385_45.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:31 proc_lsst.multi:146 [INFO]  found job 17001247 in provider checkpoint
proc_lsst.multi INFO: found job 17001247 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 235d1a310956e8b8e7da580cb66ed899 with result from task 174
parsl.dataflow.dflow DEBUG: Task 174 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3767a720-eef6-426d-9b1d-61dcc76a004e_isrForCrosstalkSources_898387_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/3767a720-eef6-426d-9b1d-61dcc76a004e_isrForCrosstalkSources_898387_59.stderr
parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac7373a8f14775b631db401cd043466 with result from task 171
parsl.dataflow.dflow DEBUG: Task 171 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/0488bae8-4d42-40cd-acd1-9aac54d1853f_isrForCrosstalkSources_898387_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/0488bae8-4d42-40cd-acd1-9aac54d1853f_isrForCrosstalkSources_898387_35.stderr
parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7dadcfe5be31fac85f7811bc5be67b30 with result from task 175
parsl.dataflow.dflow DEBUG: Task 175 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/71c9d317-bdb8-4a2b-8939-822014a5b255_isrForCrosstalkSources_898386_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/71c9d317-bdb8-4a2b-8939-822014a5b255_isrForCrosstalkSources_898386_20.stderr
parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd02994b25152380a15345f6351bb778 with result from task 164
parsl.dataflow.dflow DEBUG: Task 164 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ab2c18e9-8b74-4a1c-99cd-60a28338ca7d_isrForCrosstalkSources_898387_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ab2c18e9-8b74-4a1c-99cd-60a28338ca7d_isrForCrosstalkSources_898387_36.stderr
parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6410e22947419d199bf454b8eb3568c with result from task 177
parsl.dataflow.dflow DEBUG: Task 177 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/816524f7-38b7-463a-94a4-bd50121459dc_isrForCrosstalkSources_898281_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/816524f7-38b7-463a-94a4-bd50121459dc_isrForCrosstalkSources_898281_12.stderr
parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2384e0f88b2bbb8273d6b32ee874979c with result from task 170
parsl.dataflow.dflow DEBUG: Task 170 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e9ceb12c-396f-4d6f-9af3-625b7d4cb7f1_isrForCrosstalkSources_898385_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e9ceb12c-396f-4d6f-9af3-625b7d4cb7f1_isrForCrosstalkSources_898385_36.stderr
parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d3c43c007a6f23a7613667707583e79 with result from task 173
parsl.dataflow.dflow DEBUG: Task 173 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/2f210879-95e0-4c42-8028-7681ad7ad1a7_isrForCrosstalkSources_898385_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/2f210879-95e0-4c42-8028-7681ad7ad1a7_isrForCrosstalkSources_898385_6.stderr
parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55c231b14bcbe387928b70b04720daef with result from task 176
parsl.dataflow.dflow DEBUG: Task 176 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/3ffbb471-e6e7-4e01-b5e0-9bd7cb1e1bb1_isrForCrosstalkSources_898281_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/3ffbb471-e6e7-4e01-b5e0-9bd7cb1e1bb1_isrForCrosstalkSources_898281_11.stderr
parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4fa45a9f370d478a422c494509715c9a with result from task 120
parsl.dataflow.dflow DEBUG: Task 120 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/1b3e33d8-7946-4842-afc3-08f90357d88f_isrForCrosstalkSources_898280_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/1b3e33d8-7946-4842-afc3-08f90357d88f_isrForCrosstalkSources_898280_52.stderr
parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb22293e46ecfda6a8a67ffa95c34b84 with result from task 115
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:31 proc_lsst.multi:146 [INFO]  found job 17001248 in provider checkpoint
proc_lsst.multi INFO: found job 17001248 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow DEBUG: Task 115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/27a1bdb7-b5de-41b8-94d8-005606afd4c8_isrForCrosstalkSources_898280_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/27a1bdb7-b5de-41b8-94d8-005606afd4c8_isrForCrosstalkSources_898280_35.stderr
parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bef8afe43f044a5291ebf9cfe94ac4f8 with result from task 167
parsl.dataflow.dflow DEBUG: Task 167 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/0874b0f2-423a-4da0-9a9f-334cf0e75697_isrForCrosstalkSources_898387_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/0874b0f2-423a-4da0-9a9f-334cf0e75697_isrForCrosstalkSources_898387_57.stderr
parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84cb046ce91f5aea46ff26029f36f980 with result from task 114
parsl.dataflow.dflow DEBUG: Task 114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/1d681fb1-7a5a-4a1a-96a0-d767dc6b29b8_isrForCrosstalkSources_898280_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/1d681fb1-7a5a-4a1a-96a0-d767dc6b29b8_isrForCrosstalkSources_898280_6.stderr
parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3592fb4dc7a65d16ae68ba958ae11985 with result from task 116
parsl.dataflow.dflow DEBUG: Task 116 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5ef28667-acd8-4a08-8349-03f895b2823b_isrForCrosstalkSources_898385_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/5ef28667-acd8-4a08-8349-03f895b2823b_isrForCrosstalkSources_898385_49.stderr
parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b029a8d938c78a08097e53aca9fe6b9a with result from task 121
parsl.dataflow.dflow DEBUG: Task 121 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/b3c0a56d-7b3c-4874-8890-6c8cba4dbee0_isrForCrosstalkSources_898385_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/b3c0a56d-7b3c-4874-8890-6c8cba4dbee0_isrForCrosstalkSources_898385_22.stderr
parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d89694b73d74bbfa6e5fd33878ce8be4 with result from task 119
parsl.dataflow.dflow DEBUG: Task 119 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/db10c3e5-56bf-4a22-a702-da47f9f1f954_isrForCrosstalkSources_898281_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/db10c3e5-56bf-4a22-a702-da47f9f1f954_isrForCrosstalkSources_898281_7.stderr
parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d8656a10d7d2ec8571f150778153bb1 with result from task 117
parsl.dataflow.dflow DEBUG: Task 117 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/37cbd2c8-d0f8-4f8f-a974-cb7cd65c6612_isrForCrosstalkSources_898386_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/37cbd2c8-d0f8-4f8f-a974-cb7cd65c6612_isrForCrosstalkSources_898386_17.stderr
parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff31b6f38e5b67e41804295b3992b3ae with result from task 118
parsl.dataflow.dflow DEBUG: Task 118 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2c54af98-b1ef-4c6d-9c81-c3ad88290728_isrForCrosstalkSources_898281_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2c54af98-b1ef-4c6d-9c81-c3ad88290728_isrForCrosstalkSources_898281_62.stderr
parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c4284008f192758e49ff44c98d1d4f5b with result from task 161
parsl.dataflow.dflow DEBUG: Task 161 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ba2e802c-a399-4fe5-a11d-b88bab4a841d_isrForCrosstalkSources_898280_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ba2e802c-a399-4fe5-a11d-b88bab4a841d_isrForCrosstalkSources_898280_62.stderr
parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8a9aca783f11d8993efcb904e79865d with result from task 155
parsl.dataflow.dflow DEBUG: Task 155 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/149b2272-cb8c-4cf1-b0d7-dd93deef8bc8_isrForCrosstalkSources_898282_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/149b2272-cb8c-4cf1-b0d7-dd93deef8bc8_isrForCrosstalkSources_898282_50.stderr
parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0315fcd1a0ad9e854715a85d0b0eb8cf with result from task 154
parsl.dataflow.dflow DEBUG: Task 154 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2223fc0b-c9cc-4285-a76e-1d94426a771a_isrForCrosstalkSources_898281_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2223fc0b-c9cc-4285-a76e-1d94426a771a_isrForCrosstalkSources_898281_34.stderr
parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db838e504436555fe951748aefe3eaec with result from task 159
parsl.dataflow.dflow DEBUG: Task 159 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/8fa3d833-3e35-4eb1-8cf8-099152633dc0_isrForCrosstalkSources_898281_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/8fa3d833-3e35-4eb1-8cf8-099152633dc0_isrForCrosstalkSources_898281_41.stderr
parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 562516bf78a8177b03d2fee73a0a478e with result from task 157
parsl.dataflow.dflow DEBUG: Task 157 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/1c4641fa-d0ca-4396-8e53-9f30e0fbdeac_isrForCrosstalkSources_898386_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/1c4641fa-d0ca-4396-8e53-9f30e0fbdeac_isrForCrosstalkSources_898386_62.stderr
parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ba5f59b630b6ae56f852a8d1b302d48 with result from task 160
parsl.dataflow.dflow DEBUG: Task 160 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/86c28f90-d604-4217-8aa0-f14dfcb2fa94_isrForCrosstalkSources_898385_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/86c28f90-d604-4217-8aa0-f14dfcb2fa94_isrForCrosstalkSources_898385_48.stderr
parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 596b71009cba0cefc386a60bf17ee1f7 with result from task 158
parsl.dataflow.dflow DEBUG: Task 158 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/289f3346-fc10-457d-8055-6ab9ed26ef78_isrForCrosstalkSources_898387_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/289f3346-fc10-457d-8055-6ab9ed26ef78_isrForCrosstalkSources_898387_51.stderr
parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eedcc0169bcca30c9666bffeb7c231e9 with result from task 156
parsl.dataflow.dflow DEBUG: Task 156 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/768f2d09-f2d6-4598-b968-1add7587a67c_isrForCrosstalkSources_898281_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/768f2d09-f2d6-4598-b968-1add7587a67c_isrForCrosstalkSources_898281_23.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:32 proc_lsst.multi:146 [INFO]  found job 17001249 in provider checkpoint
proc_lsst.multi INFO: found job 17001249 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 190
parsl.dataflow.strategy DEBUG: Executor multi has 190 active tasks, 38/55 running/pending blocks, and 298 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 190
parsl.dataflow.strategy DEBUG: Executor multi has 190 active tasks, 38/55 running/pending blocks, and 298 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry edb1344cb881e45cbfa49bebc91b1c35 with result from task 182
parsl.dataflow.dflow DEBUG: Task 182 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/4fc96d1d-deb2-47d0-a7cc-4ccf73538d43_isrForCrosstalkSources_898385_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/4fc96d1d-deb2-47d0-a7cc-4ccf73538d43_isrForCrosstalkSources_898385_51.stderr
parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41ab9c629796a2c08fe942d5081dd80e with result from task 191
parsl.dataflow.dflow DEBUG: Task 191 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2eb96c39-96b6-4542-a3a1-ada8443ee1eb_isrForCrosstalkSources_898386_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2eb96c39-96b6-4542-a3a1-ada8443ee1eb_isrForCrosstalkSources_898386_2.stderr
parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 515d666e2887afb720e9d531f31c5741 with result from task 325
parsl.dataflow.dflow DEBUG: Task 325 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/630ab270-f1f7-4257-b7a2-871b4701ec1c_isrForCrosstalkSources_898385_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/630ab270-f1f7-4257-b7a2-871b4701ec1c_isrForCrosstalkSources_898385_26.stderr
parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53f1b7f796af8389d5db2eb3d6a1ac30 with result from task 316
parsl.dataflow.dflow DEBUG: Task 316 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/5b4aa234-11c1-4e83-96d4-e0c163055295_isrForCrosstalkSources_898282_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/5b4aa234-11c1-4e83-96d4-e0c163055295_isrForCrosstalkSources_898282_18.stderr
parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a71cfcc6d7f00279495249626a8220a9 with result from task 317
parsl.dataflow.dflow DEBUG: Task 317 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d21fc3b7-8822-4f7b-9a2e-22fa43bdaaba_isrForCrosstalkSources_898282_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d21fc3b7-8822-4f7b-9a2e-22fa43bdaaba_isrForCrosstalkSources_898282_54.stderr
parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f4801836c54f3fd23e8e3f815c6c4a0 with result from task 319
parsl.dataflow.dflow DEBUG: Task 319 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/53ae2ec5-0291-48f2-9ad5-98415b27c72e_isrForCrosstalkSources_898281_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/53ae2ec5-0291-48f2-9ad5-98415b27c72e_isrForCrosstalkSources_898281_6.stderr
parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 76e3399a8eac7c09a4f1560511ba6594 with result from task 256
parsl.dataflow.dflow DEBUG: Task 256 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bdc212cb-d55f-483f-932c-c87ebe523b16_isrForCrosstalkSources_898281_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bdc212cb-d55f-483f-932c-c87ebe523b16_isrForCrosstalkSources_898281_22.stderr
parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 576217ea7337789d599520895db87e9e with result from task 320
parsl.dataflow.dflow DEBUG: Task 320 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/607401a1-a2dc-4f23-9506-74cfa81125b3_isrForCrosstalkSources_898386_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/607401a1-a2dc-4f23-9506-74cfa81125b3_isrForCrosstalkSources_898386_8.stderr
parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry be129f901e0490b0266f4a6b44c8a265 with result from task 324
parsl.dataflow.dflow DEBUG: Task 324 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c4014a47-0d18-4c3e-b5b4-0c5b6c7a44cb_isrForCrosstalkSources_898282_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c4014a47-0d18-4c3e-b5b4-0c5b6c7a44cb_isrForCrosstalkSources_898282_55.stderr
parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e2d2547732936dcc33165c91c11a2fce with result from task 327
parsl.dataflow.dflow DEBUG: Task 327 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/e77856db-68f5-44b1-a1b2-855a5e9d112d_isrForCrosstalkSources_898387_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/e77856db-68f5-44b1-a1b2-855a5e9d112d_isrForCrosstalkSources_898387_27.stderr
parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9868dd2b2a8b8e13520124d625c8d08e with result from task 322
parsl.dataflow.dflow DEBUG: Task 322 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6c8ae889-8547-4c3f-b7d7-85b13f8fb4d4_isrForCrosstalkSources_898385_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6c8ae889-8547-4c3f-b7d7-85b13f8fb4d4_isrForCrosstalkSources_898385_40.stderr
parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b252bf0be9e5af9db623cb4d72a6d32e with result from task 359
parsl.dataflow.dflow DEBUG: Task 359 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/37d1be8e-4765-41e1-9f27-047ede4b58ae_isrForCrosstalkSources_898281_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/37d1be8e-4765-41e1-9f27-047ede4b58ae_isrForCrosstalkSources_898281_5.stderr
parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f259640ad7e8f07375e6e3615b6cd650 with result from task 315
parsl.dataflow.dflow DEBUG: Task 315 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b49ca4f0-9493-4359-aeb1-60d8478ad3c1_isrForCrosstalkSources_898280_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b49ca4f0-9493-4359-aeb1-60d8478ad3c1_isrForCrosstalkSources_898280_15.stderr
parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0d148ffe53b155c138bc652136d1714 with result from task 258
parsl.dataflow.dflow DEBUG: Task 258 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/5f0062c5-6e1a-435a-89a1-c224fc7d4a5a_isrForCrosstalkSources_898280_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/5f0062c5-6e1a-435a-89a1-c224fc7d4a5a_isrForCrosstalkSources_898280_36.stderr
parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 837a0ee800c66ebfe852b36cbbd4100d with result from task 257
parsl.dataflow.dflow DEBUG: Task 257 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e031157f-725f-4e41-adb6-4cb744f312fd_isrForCrosstalkSources_898280_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e031157f-725f-4e41-adb6-4cb744f312fd_isrForCrosstalkSources_898280_2.stderr
parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c1d8386b39509c16f0d25ecbb2a5fda with result from task 318
parsl.dataflow.dflow DEBUG: Task 318 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/c1d2c9ab-8113-47ce-866e-ee4b8405c02a_isrForCrosstalkSources_898387_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/c1d2c9ab-8113-47ce-866e-ee4b8405c02a_isrForCrosstalkSources_898387_31.stderr
parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 29bd78caa196412844b8150f3e6e629d with result from task 321
parsl.dataflow.dflow DEBUG: Task 321 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/751c9502-a4bf-479e-b516-5352c44e09d7_isrForCrosstalkSources_898385_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/751c9502-a4bf-479e-b516-5352c44e09d7_isrForCrosstalkSources_898385_41.stderr
parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e9220faef8ae5573d4bc6bf51169175 with result from task 323
parsl.dataflow.dflow DEBUG: Task 323 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/0200c459-0947-45b4-bd4f-9ac08da34147_isrForCrosstalkSources_898385_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/0200c459-0947-45b4-bd4f-9ac08da34147_isrForCrosstalkSources_898385_44.stderr
parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccc36667a952f08f5b8b8dea3afc0044 with result from task 335
parsl.dataflow.dflow DEBUG: Task 335 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/40a664ac-783b-4169-850f-1c2c48511bcf_isrForCrosstalkSources_898387_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/40a664ac-783b-4169-850f-1c2c48511bcf_isrForCrosstalkSources_898387_19.stderr
parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1c69a12b9d3d4ceb34e0c807169f0144 with result from task 326
parsl.dataflow.dflow DEBUG: Task 326 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4bbd3933-ff5c-4285-849b-6f0f97bd82b3_isrForCrosstalkSources_898281_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4bbd3933-ff5c-4285-849b-6f0f97bd82b3_isrForCrosstalkSources_898281_52.stderr
2024-03-11 23:02:37 proc_lsst.multi:146 [INFO]  found job 26021 in provider local
proc_lsst.multi INFO: found job 26021 in provider local
parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0736e63cbd2ca6e9ef6ec68dfa31fc97 with result from task 356
parsl.dataflow.dflow DEBUG: Task 356 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b55360d3-afee-4e6d-9752-28510fa8d426_isrForCrosstalkSources_898280_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b55360d3-afee-4e6d-9752-28510fa8d426_isrForCrosstalkSources_898280_4.stderr
parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c6893864d951a51c6767bd76bb8a651 with result from task 355
parsl.dataflow.dflow DEBUG: Task 355 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b4d9aef6-3521-4525-8db0-81933032165f_isrForCrosstalkSources_898280_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b4d9aef6-3521-4525-8db0-81933032165f_isrForCrosstalkSources_898280_42.stderr
2024-03-11 23:02:37 proc_lsst.multi:146 [INFO]  found job 17001157 in provider astro
proc_lsst.multi INFO: found job 17001157 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f4d8a9c99507f208af38674db1af012 with result from task 345
parsl.dataflow.dflow DEBUG: Task 345 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/720d663e-f23b-4a72-82cd-a5f9a6cda3ce_isrForCrosstalkSources_898387_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/720d663e-f23b-4a72-82cd-a5f9a6cda3ce_isrForCrosstalkSources_898387_1.stderr
parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87e6e3bdeddf6e18ce1953b4ff8c76ab with result from task 350
parsl.dataflow.dflow DEBUG: Task 350 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7226e17e-3565-4c14-a903-42eddc920ee3_isrForCrosstalkSources_898387_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7226e17e-3565-4c14-a903-42eddc920ee3_isrForCrosstalkSources_898387_28.stderr
parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a0f72b4a1e951ab353972e0f175552f with result from task 349
parsl.dataflow.dflow DEBUG: Task 349 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/429c661b-6079-4f16-90b5-051fa2fdbed1_isrForCrosstalkSources_898387_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/429c661b-6079-4f16-90b5-051fa2fdbed1_isrForCrosstalkSources_898387_20.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:37 proc_lsst.multi:146 [INFO]  found job 17001158 in provider astro
proc_lsst.multi INFO: found job 17001158 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd18f2619c4de304eec103ab173f6ecb with result from task 358
parsl.dataflow.dflow DEBUG: Task 358 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/2815bdd4-8a9f-4236-9d42-897deb77c605_isrForCrosstalkSources_898387_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/2815bdd4-8a9f-4236-9d42-897deb77c605_isrForCrosstalkSources_898387_29.stderr
parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 614af394068f163ef09c0fdeaf4948cf with result from task 347
parsl.dataflow.dflow DEBUG: Task 347 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3083d8ef-7457-446f-821d-f2a157e6ac07_isrForCrosstalkSources_898385_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/3083d8ef-7457-446f-821d-f2a157e6ac07_isrForCrosstalkSources_898385_32.stderr
parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e50ed825a4d122322335a7d79adecd6 with result from task 333
parsl.dataflow.dflow DEBUG: Task 333 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/9918d6e0-2cc5-4c13-8e6c-57f79fec3663_isrForCrosstalkSources_898386_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/9918d6e0-2cc5-4c13-8e6c-57f79fec3663_isrForCrosstalkSources_898386_13.stderr
parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20d7caf24708403134fbd2baf2f103ec with result from task 339
parsl.dataflow.dflow DEBUG: Task 339 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/242a9ce4-d521-4ba1-a20f-d2a75e32523d_isrForCrosstalkSources_898386_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/242a9ce4-d521-4ba1-a20f-d2a75e32523d_isrForCrosstalkSources_898386_5.stderr
parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d84fa0ab8216a63e06c61b5fd97a6e79 with result from task 336
parsl.dataflow.dflow DEBUG: Task 336 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a5b11818-3de9-44b8-aa9c-04259c68e947_isrForCrosstalkSources_898385_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a5b11818-3de9-44b8-aa9c-04259c68e947_isrForCrosstalkSources_898385_17.stderr
parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb0818ed866f942e7218457e7593dd40 with result from task 361
parsl.dataflow.dflow DEBUG: Task 361 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c7154d1e-c429-4841-b43f-0c7b5ac48d99_isrForCrosstalkSources_898280_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/c7154d1e-c429-4841-b43f-0c7b5ac48d99_isrForCrosstalkSources_898280_34.stderr
parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bafbf13543f12a44874a6cfba97b2204 with result from task 346
parsl.dataflow.dflow DEBUG: Task 346 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/027491eb-7356-44c6-843b-96a99238f955_isrForCrosstalkSources_898281_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/027491eb-7356-44c6-843b-96a99238f955_isrForCrosstalkSources_898281_2.stderr
parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 74b0aaa2e07760e09868dd7800f3afde with result from task 348
parsl.dataflow.dflow DEBUG: Task 348 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/11304bca-6b73-4236-910e-1b2c68e4cdd9_isrForCrosstalkSources_898282_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/11304bca-6b73-4236-910e-1b2c68e4cdd9_isrForCrosstalkSources_898282_8.stderr
parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e24ed6d767b77025cf5a7caba6f5110 with result from task 344
parsl.dataflow.dflow DEBUG: Task 344 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1aa518c5-7761-426f-8c65-540f6ff37bf9_isrForCrosstalkSources_898282_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/1aa518c5-7761-426f-8c65-540f6ff37bf9_isrForCrosstalkSources_898282_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:37 proc_lsst.multi:146 [INFO]  found job 17001159 in provider astro
proc_lsst.multi INFO: found job 17001159 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 641159561e594446662b413759c2dc6d with result from task 371
parsl.dataflow.dflow DEBUG: Task 371 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/5918aaa5-c014-4721-bf6b-d623ce0496d7_isrForCrosstalkSources_898386_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/5918aaa5-c014-4721-bf6b-d623ce0496d7_isrForCrosstalkSources_898386_34.stderr
parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e30d6738a72a90318693aa7af2a32a5e with result from task 332
parsl.dataflow.dflow DEBUG: Task 332 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4393031c-16a2-4531-a913-3f19277e357a_isrForCrosstalkSources_898282_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4393031c-16a2-4531-a913-3f19277e357a_isrForCrosstalkSources_898282_15.stderr
parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e39a0c32e43f9a31edcb06551d49988f with result from task 351
parsl.dataflow.dflow DEBUG: Task 351 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/34245fbc-5a96-4b96-ac81-bbac51cbcae3_isrForCrosstalkSources_898281_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/34245fbc-5a96-4b96-ac81-bbac51cbcae3_isrForCrosstalkSources_898281_37.stderr
parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0959e05bac64513bcbbb49b0bbe5052e with result from task 342
parsl.dataflow.dflow DEBUG: Task 342 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b190115a-6023-45d9-8086-f4d91f66f070_isrForCrosstalkSources_898280_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b190115a-6023-45d9-8086-f4d91f66f070_isrForCrosstalkSources_898280_19.stderr
parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 56f0d9bc164cb567197e1c2f7809ad61 with result from task 352
parsl.dataflow.dflow DEBUG: Task 352 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ac712d0a-5231-4bb6-a624-86d8250609c4_isrForCrosstalkSources_898280_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ac712d0a-5231-4bb6-a624-86d8250609c4_isrForCrosstalkSources_898280_41.stderr
parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b523b9ba150011ba7dacbfb90ac8cd2d with result from task 338
parsl.dataflow.dflow DEBUG: Task 338 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bab1cba7-112f-4d0b-86ab-2272f2372eaf_isrForCrosstalkSources_898282_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bab1cba7-112f-4d0b-86ab-2272f2372eaf_isrForCrosstalkSources_898282_9.stderr
parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4174c9352fe44ba278d8d08611455fc2 with result from task 364
parsl.dataflow.dflow DEBUG: Task 364 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c568fa43-2990-4607-a419-7e51d81abf78_isrForCrosstalkSources_898282_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c568fa43-2990-4607-a419-7e51d81abf78_isrForCrosstalkSources_898282_11.stderr
parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6887d46c81304dbad9a27226bb7d832d with result from task 354
parsl.dataflow.dflow DEBUG: Task 354 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/aa6feb82-077d-4d2b-baf5-e9dfb6363e23_isrForCrosstalkSources_898282_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/aa6feb82-077d-4d2b-baf5-e9dfb6363e23_isrForCrosstalkSources_898282_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:38 proc_lsst.multi:146 [INFO]  found job 17001160 in provider astro
proc_lsst.multi INFO: found job 17001160 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 82c93a12f44e621ce97e3afda5554e6b with result from task 341
parsl.dataflow.dflow DEBUG: Task 341 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/5ebf063f-31a2-4381-acef-ab567cfc6b42_isrForCrosstalkSources_898281_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/5ebf063f-31a2-4381-acef-ab567cfc6b42_isrForCrosstalkSources_898281_35.stderr
parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 345cb3fe5d8c65d87aa7fc75f156e20c with result from task 360
parsl.dataflow.dflow DEBUG: Task 360 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/558166c5-afdb-4318-844b-759f79186eef_isrForCrosstalkSources_898386_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/558166c5-afdb-4318-844b-759f79186eef_isrForCrosstalkSources_898386_27.stderr
parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 501f5a341f224040727570a37419fd69 with result from task 362
parsl.dataflow.dflow DEBUG: Task 362 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7ebc477f-cfb6-401f-8e44-fa8955a3d6c4_isrForCrosstalkSources_898387_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/7ebc477f-cfb6-401f-8e44-fa8955a3d6c4_isrForCrosstalkSources_898387_34.stderr
parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a0ce51df469169ab60699fc2586173b with result from task 357
parsl.dataflow.dflow DEBUG: Task 357 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/07f758e8-ff48-47e5-8e81-39a531233093_isrForCrosstalkSources_898282_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/07f758e8-ff48-47e5-8e81-39a531233093_isrForCrosstalkSources_898282_44.stderr
parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d893bcb42cce18929bf63858b5e3247 with result from task 366
parsl.dataflow.dflow DEBUG: Task 366 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/9a93525c-ce44-4f0d-845d-5293befd5f56_isrForCrosstalkSources_898280_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/9a93525c-ce44-4f0d-845d-5293befd5f56_isrForCrosstalkSources_898280_27.stderr
parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7be1ddfde58798a19ce233177bf2638 with result from task 343
parsl.dataflow.dflow DEBUG: Task 343 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/6c529368-b961-4b06-9a14-883c8585071a_isrForCrosstalkSources_898281_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/6c529368-b961-4b06-9a14-883c8585071a_isrForCrosstalkSources_898281_53.stderr
parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b5974477791d632c1f9850f12b828e9 with result from task 370
parsl.dataflow.dflow DEBUG: Task 370 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/06ca93a3-7940-480e-870f-24ba94a8ff84_isrForCrosstalkSources_898385_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/06ca93a3-7940-480e-870f-24ba94a8ff84_isrForCrosstalkSources_898385_54.stderr
parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8459a0a91c7037567c341b8f0a31732c with result from task 337
parsl.dataflow.dflow DEBUG: Task 337 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/030be73a-4818-4936-b25c-8adf17812d7f_isrForCrosstalkSources_898387_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/030be73a-4818-4936-b25c-8adf17812d7f_isrForCrosstalkSources_898387_22.stderr
parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2738386bff08c925be9c408936082ed7 with result from task 340
parsl.dataflow.dflow DEBUG: Task 340 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4f98c16d-8a2c-4fb7-a6df-cffed4130194_isrForCrosstalkSources_898281_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4f98c16d-8a2c-4fb7-a6df-cffed4130194_isrForCrosstalkSources_898281_57.stderr
parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ae59b860807fe1a9f54e03404284e6ab with result from task 363
parsl.dataflow.dflow DEBUG: Task 363 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/42d6ed7f-4bae-45c4-819c-6b26fc9c39b1_isrForCrosstalkSources_898387_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/42d6ed7f-4bae-45c4-819c-6b26fc9c39b1_isrForCrosstalkSources_898387_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:38 proc_lsst.multi:146 [INFO]  found job 17001161 in provider astro
proc_lsst.multi INFO: found job 17001161 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 54c2c4da9b2fb5f7deb519f00efaa8d4 with result from task 368
parsl.dataflow.dflow DEBUG: Task 368 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7afb4b26-8028-4f5d-b669-295e7f2064e6_isrForCrosstalkSources_898281_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7afb4b26-8028-4f5d-b669-295e7f2064e6_isrForCrosstalkSources_898281_32.stderr
parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ec75560d0527e356175cb0830e23272 with result from task 334
parsl.dataflow.dflow DEBUG: Task 334 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/4a28f157-388f-4f0d-92a0-1ae9ca750ea9_isrForCrosstalkSources_898280_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/4a28f157-388f-4f0d-92a0-1ae9ca750ea9_isrForCrosstalkSources_898280_10.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:38 proc_lsst.multi:146 [INFO]  found job 17001162 in provider astro
proc_lsst.multi INFO: found job 17001162 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee571742c7c94f3471948c33998072f6 with result from task 369
parsl.dataflow.dflow DEBUG: Task 369 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e174afc5-e8ae-47a8-9e09-df333ac75a7a_isrForCrosstalkSources_898280_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e174afc5-e8ae-47a8-9e09-df333ac75a7a_isrForCrosstalkSources_898280_58.stderr
parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f8b3542cf3a4605ebd9f8841d91483b7 with result from task 353
parsl.dataflow.dflow DEBUG: Task 353 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b98fb9f3-7db6-4ded-979f-ab27ed74880c_isrForCrosstalkSources_898280_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b98fb9f3-7db6-4ded-979f-ab27ed74880c_isrForCrosstalkSources_898280_28.stderr
parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 924e58243137bb669dcaebc287cf4242 with result from task 365
parsl.dataflow.dflow DEBUG: Task 365 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/89c839f5-00de-485a-9fa6-05a244de14ef_isrForCrosstalkSources_898386_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/89c839f5-00de-485a-9fa6-05a244de14ef_isrForCrosstalkSources_898386_6.stderr
parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 56cba9d7b023e9eeff7c90e6239dd3d2 with result from task 328
parsl.dataflow.dflow DEBUG: Task 328 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/415f4bec-3d69-4046-90ac-2cb1ed6dfcbf_isrForCrosstalkSources_898386_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/415f4bec-3d69-4046-90ac-2cb1ed6dfcbf_isrForCrosstalkSources_898386_59.stderr
parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 75775952474ca2810ad2db57bde11337 with result from task 329
parsl.dataflow.dflow DEBUG: Task 329 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/628146a3-9b1b-403e-9f3f-3fdee99e6cf8_isrForCrosstalkSources_898386_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/628146a3-9b1b-403e-9f3f-3fdee99e6cf8_isrForCrosstalkSources_898386_55.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:38 proc_lsst.multi:146 [INFO]  found job 17001163 in provider astro
proc_lsst.multi INFO: found job 17001163 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37a3b264f5e34ae774945abc2e3e71f7 with result from task 331
parsl.dataflow.dflow DEBUG: Task 331 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b32a9a92-4e2d-4d89-add2-88eea440b292_isrForCrosstalkSources_898282_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b32a9a92-4e2d-4d89-add2-88eea440b292_isrForCrosstalkSources_898282_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:38 proc_lsst.multi:146 [INFO]  found job 17001164 in provider astro
proc_lsst.multi INFO: found job 17001164 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2cc21860577ef9a3dde3e11285520b36 with result from task 330
parsl.dataflow.dflow DEBUG: Task 330 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/be6ac907-e2df-41f4-802a-fca91996f735_isrForCrosstalkSources_898385_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/be6ac907-e2df-41f4-802a-fca91996f735_isrForCrosstalkSources_898385_58.stderr
parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b559ba77febffa30329b6cf61b71774d with result from task 367
parsl.dataflow.dflow DEBUG: Task 367 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/3b2f50cb-d647-489d-b83a-6e7e142be2ee_isrForCrosstalkSources_898386_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/3b2f50cb-d647-489d-b83a-6e7e142be2ee_isrForCrosstalkSources_898386_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:39 proc_lsst.multi:146 [INFO]  found job 17001165 in provider astro
proc_lsst.multi INFO: found job 17001165 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:39 proc_lsst.multi:146 [INFO]  found job 17001166 in provider astro
proc_lsst.multi INFO: found job 17001166 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:39 proc_lsst.multi:146 [INFO]  found job 17001167 in provider astro
proc_lsst.multi INFO: found job 17001167 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:39 proc_lsst.multi:146 [INFO]  found job 17001168 in provider astro
proc_lsst.multi INFO: found job 17001168 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:39 proc_lsst.multi:146 [INFO]  found job 17001169 in provider astro
proc_lsst.multi INFO: found job 17001169 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:39 proc_lsst.multi:146 [INFO]  found job 17001170 in provider astro
proc_lsst.multi INFO: found job 17001170 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:40 proc_lsst.multi:146 [INFO]  found job 17001171 in provider astro
proc_lsst.multi INFO: found job 17001171 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:40 proc_lsst.multi:146 [INFO]  found job 17001172 in provider astro
proc_lsst.multi INFO: found job 17001172 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:40 proc_lsst.multi:146 [INFO]  found job 17001173 in provider astro
proc_lsst.multi INFO: found job 17001173 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:40 proc_lsst.multi:146 [INFO]  found job 17001174 in provider astro
proc_lsst.multi INFO: found job 17001174 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:40 proc_lsst.multi:146 [INFO]  found job 17001175 in provider astro
proc_lsst.multi INFO: found job 17001175 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:40 proc_lsst.multi:146 [INFO]  found job 17001176 in provider astro
proc_lsst.multi INFO: found job 17001176 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:41 proc_lsst.multi:146 [INFO]  found job 17001177 in provider astro
proc_lsst.multi INFO: found job 17001177 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:41 proc_lsst.multi:146 [INFO]  found job 17001178 in provider astro
proc_lsst.multi INFO: found job 17001178 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:41 proc_lsst.multi:146 [INFO]  found job 17001179 in provider astro
proc_lsst.multi INFO: found job 17001179 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:41 proc_lsst.multi:146 [INFO]  found job 17001180 in provider astro
proc_lsst.multi INFO: found job 17001180 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:41 proc_lsst.multi:146 [INFO]  found job 17001181 in provider astro
proc_lsst.multi INFO: found job 17001181 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:41 proc_lsst.multi:146 [INFO]  found job 17001182 in provider astro
proc_lsst.multi INFO: found job 17001182 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:42 proc_lsst.multi:146 [INFO]  found job 17001183 in provider astro
proc_lsst.multi INFO: found job 17001183 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:42 proc_lsst.multi:146 [INFO]  found job 17001184 in provider astro
proc_lsst.multi INFO: found job 17001184 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:42 proc_lsst.multi:146 [INFO]  found job 17001185 in provider astro
proc_lsst.multi INFO: found job 17001185 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:42 proc_lsst.multi:146 [INFO]  found job 17001186 in provider astro
proc_lsst.multi INFO: found job 17001186 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001157,17001158,17001159,17001160,17001161,17001162,17001163,17001164,17001165,17001166,17001167,17001168,17001169,17001170,17001171,17001172,17001173,17001174,17001175,17001176,17001177,17001178,17001179,17001180,17001181,17001182,17001183,17001184,17001185,17001186'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001157 PD
17001158 PD
17001159 PD
17001160 PD
17001161 PD
17001162 PD
17001163 PD
17001164 PD
17001165 PD
17001166 PD
17001167 PD
17001168 PD
17001169 PD
17001170 PD
17001171 PD
17001172 PD
17001173 PD
17001174 PD
17001175 PD
17001176 PD
17001177 PD
17001178 PD
17001179 PD
17001180 PD
17001181 PD
17001182 PD
17001183 PD
17001184 PD
17001185 PD
17001186 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001157 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001158 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001159 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001160 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001161 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001162 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001163 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001164 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001165 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001166 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001167 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001168 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001169 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001170 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001171 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001172 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001173 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001174 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001175 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001176 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001177 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001178 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001179 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001180 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001181 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001182 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001183 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001184 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001185 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001186 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:42 proc_lsst.multi:146 [INFO]  found job 17001187 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001187 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:43 proc_lsst.multi:146 [INFO]  found job 17001188 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001188 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:43 proc_lsst.multi:146 [INFO]  found job 17001189 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001189 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:43 proc_lsst.multi:146 [INFO]  found job 17001190 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001190 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:43 proc_lsst.multi:146 [INFO]  found job 17001191 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001191 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001187,17001188,17001189,17001190,17001191'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001187 PD
17001188 PD
17001189 PD
17001190 PD
17001191 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001187 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001188 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001189 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001190 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001191 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:43 proc_lsst.multi:146 [INFO]  found job 17001192 in provider escience_a40
proc_lsst.multi INFO: found job 17001192 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:43 proc_lsst.multi:146 [INFO]  found job 17001193 in provider escience_a40
proc_lsst.multi INFO: found job 17001193 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:44 proc_lsst.multi:146 [INFO]  found job 17001194 in provider escience_a40
proc_lsst.multi INFO: found job 17001194 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:44 proc_lsst.multi:146 [INFO]  found job 17001195 in provider escience_a40
proc_lsst.multi INFO: found job 17001195 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:44 proc_lsst.multi:146 [INFO]  found job 17001196 in provider escience_a40
proc_lsst.multi INFO: found job 17001196 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:44 proc_lsst.multi:146 [INFO]  found job 17001197 in provider escience_a40
proc_lsst.multi INFO: found job 17001197 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:44 proc_lsst.multi:146 [INFO]  found job 17001198 in provider escience_a40
proc_lsst.multi INFO: found job 17001198 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:44 proc_lsst.multi:146 [INFO]  found job 17001199 in provider escience_a40
proc_lsst.multi INFO: found job 17001199 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:45 proc_lsst.multi:146 [INFO]  found job 17001200 in provider escience_a40
proc_lsst.multi INFO: found job 17001200 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:45 proc_lsst.multi:146 [INFO]  found job 17001201 in provider escience_a40
proc_lsst.multi INFO: found job 17001201 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:45 proc_lsst.multi:146 [INFO]  found job 17001202 in provider escience_a40
proc_lsst.multi INFO: found job 17001202 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:45 proc_lsst.multi:146 [INFO]  found job 17001203 in provider escience_a40
proc_lsst.multi INFO: found job 17001203 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:45 proc_lsst.multi:146 [INFO]  found job 17001204 in provider escience_a40
proc_lsst.multi INFO: found job 17001204 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:46 proc_lsst.multi:146 [INFO]  found job 17001205 in provider escience_a40
proc_lsst.multi INFO: found job 17001205 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:46 proc_lsst.multi:146 [INFO]  found job 17001206 in provider escience_a40
proc_lsst.multi INFO: found job 17001206 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:46 proc_lsst.multi:146 [INFO]  found job 17001207 in provider escience_a40
proc_lsst.multi INFO: found job 17001207 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:46 proc_lsst.multi:146 [INFO]  found job 17001208 in provider escience_a40
proc_lsst.multi INFO: found job 17001208 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:46 proc_lsst.multi:146 [INFO]  found job 17001209 in provider escience_a40
proc_lsst.multi INFO: found job 17001209 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:46 proc_lsst.multi:146 [INFO]  found job 17001210 in provider escience_a40
proc_lsst.multi INFO: found job 17001210 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:47 proc_lsst.multi:146 [INFO]  found job 17001211 in provider escience_a40
proc_lsst.multi INFO: found job 17001211 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001192,17001193,17001194,17001195,17001196,17001197,17001198,17001199,17001200,17001201,17001202,17001203,17001204,17001205,17001206,17001207,17001208,17001209,17001210,17001211'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001192 PD
17001193 PD
17001194 PD
17001195 PD
17001196 PD
17001197 PD
17001198 PD
17001199 PD
17001200 PD
17001201 PD
17001202 PD
17001203 PD
17001204 PD
17001205 PD
17001206 PD
17001207 PD
17001208 PD
17001209 PD
17001210 PD
17001211 PD
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001192 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001193 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001194 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001195 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001196 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001197 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001198 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001199 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001200 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001201 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001202 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001203 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001204 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001205 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001206 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001207 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001208 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001209 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001210 with slurm status PD to parsl state JobState.PENDING
parsl.providers.slurm.slurm DEBUG: Updating job 17001211 with slurm status PD to parsl state JobState.PENDING
2024-03-11 23:02:47 proc_lsst.multi:146 [INFO]  found job 17001212 in provider checkpoint
proc_lsst.multi INFO: found job 17001212 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:47 proc_lsst.multi:146 [INFO]  found job 17001213 in provider checkpoint
proc_lsst.multi INFO: found job 17001213 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:47 proc_lsst.multi:146 [INFO]  found job 17001214 in provider checkpoint
proc_lsst.multi INFO: found job 17001214 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:47 proc_lsst.multi:146 [INFO]  found job 17001215 in provider checkpoint
proc_lsst.multi INFO: found job 17001215 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:47 proc_lsst.multi:146 [INFO]  found job 17001216 in provider checkpoint
proc_lsst.multi INFO: found job 17001216 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:48 proc_lsst.multi:146 [INFO]  found job 17001217 in provider checkpoint
proc_lsst.multi INFO: found job 17001217 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:48 proc_lsst.multi:146 [INFO]  found job 17001218 in provider checkpoint
proc_lsst.multi INFO: found job 17001218 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:48 proc_lsst.multi:146 [INFO]  found job 17001219 in provider checkpoint
proc_lsst.multi INFO: found job 17001219 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:48 proc_lsst.multi:146 [INFO]  found job 17001220 in provider checkpoint
proc_lsst.multi INFO: found job 17001220 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:48 proc_lsst.multi:146 [INFO]  found job 17001221 in provider checkpoint
proc_lsst.multi INFO: found job 17001221 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:48 proc_lsst.multi:146 [INFO]  found job 17001222 in provider checkpoint
proc_lsst.multi INFO: found job 17001222 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:49 proc_lsst.multi:146 [INFO]  found job 17001223 in provider checkpoint
proc_lsst.multi INFO: found job 17001223 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1bd654a0f3731e00b487675e6a93419b with result from task 202
parsl.dataflow.dflow DEBUG: Task 202 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/db914e92-dab1-4410-90d6-3d938160d37b_isrForCrosstalkSources_898281_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/db914e92-dab1-4410-90d6-3d938160d37b_isrForCrosstalkSources_898281_25.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:49 proc_lsst.multi:146 [INFO]  found job 17001224 in provider checkpoint
proc_lsst.multi INFO: found job 17001224 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:49 proc_lsst.multi:146 [INFO]  found job 17001225 in provider checkpoint
proc_lsst.multi INFO: found job 17001225 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4274bb769df40882a09890e78bce5e46 with result from task 206
parsl.dataflow.dflow DEBUG: Task 206 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/669c7604-2662-488d-b798-ea29d7326fbe_isrForCrosstalkSources_898385_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/669c7604-2662-488d-b798-ea29d7326fbe_isrForCrosstalkSources_898385_21.stderr
parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fd87fb449f06754074b37b25e3249747 with result from task 200
parsl.dataflow.dflow DEBUG: Task 200 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/a8ff940c-31ac-456e-b2c2-1323dff86095_isrForCrosstalkSources_898282_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/a8ff940c-31ac-456e-b2c2-1323dff86095_isrForCrosstalkSources_898282_32.stderr
parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 117db7fe8de6c022c858d1883f88c467 with result from task 207
parsl.dataflow.dflow DEBUG: Task 207 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/4df39916-772c-4bee-99f0-2fe63d6c2cc0_isrForCrosstalkSources_898280_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/4df39916-772c-4bee-99f0-2fe63d6c2cc0_isrForCrosstalkSources_898280_18.stderr
parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a28c15290355aa82aee6d6d88d52dab9 with result from task 205
parsl.dataflow.dflow DEBUG: Task 205 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/f294daec-5f8c-4c96-b1e8-c83eb9a2f74f_isrForCrosstalkSources_898385_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/f294daec-5f8c-4c96-b1e8-c83eb9a2f74f_isrForCrosstalkSources_898385_20.stderr
parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a8fc9c81555a484d01d00e1a96e67d0 with result from task 203
parsl.dataflow.dflow DEBUG: Task 203 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/95bc3c06-7e98-43e6-9bb0-344f321744e3_isrForCrosstalkSources_898387_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/95bc3c06-7e98-43e6-9bb0-344f321744e3_isrForCrosstalkSources_898387_43.stderr
parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a34415b88be3bcd2d8974a8e0f65bb3e with result from task 201
parsl.dataflow.dflow DEBUG: Task 201 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/00d086d7-9044-4680-b783-a13bc302a030_isrForCrosstalkSources_898386_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/00d086d7-9044-4680-b783-a13bc302a030_isrForCrosstalkSources_898386_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:49 proc_lsst.multi:146 [INFO]  found job 17001226 in provider checkpoint
proc_lsst.multi INFO: found job 17001226 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d5276aca2bcc7e2a3ad865aa08b1a4d with result from task 204
parsl.dataflow.dflow DEBUG: Task 204 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/e086d7df-5ef8-4b4f-b828-18cf3c930989_isrForCrosstalkSources_898386_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/e086d7df-5ef8-4b4f-b828-18cf3c930989_isrForCrosstalkSources_898386_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:49 proc_lsst.multi:146 [INFO]  found job 17001227 in provider checkpoint
proc_lsst.multi INFO: found job 17001227 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:49 proc_lsst.multi:146 [INFO]  found job 17001228 in provider checkpoint
proc_lsst.multi INFO: found job 17001228 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:50 proc_lsst.multi:146 [INFO]  found job 17001229 in provider checkpoint
proc_lsst.multi INFO: found job 17001229 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:50 proc_lsst.multi:146 [INFO]  found job 17001230 in provider checkpoint
proc_lsst.multi INFO: found job 17001230 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0643190ea6be4cbe7f1032b6940dcf74 with result from task 199
parsl.dataflow.dflow DEBUG: Task 199 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bb0f4037-073e-4caa-8183-f7adf46b46a5_isrForCrosstalkSources_898281_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bb0f4037-073e-4caa-8183-f7adf46b46a5_isrForCrosstalkSources_898281_8.stderr
parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2944c622f023f8226f5fce14add07361 with result from task 193
parsl.dataflow.dflow DEBUG: Task 193 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/a1e6097a-da00-47a2-b5d3-f9424178eed5_isrForCrosstalkSources_898281_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/a1e6097a-da00-47a2-b5d3-f9424178eed5_isrForCrosstalkSources_898281_39.stderr
parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 252340169dfa662843ab667174b30ad5 with result from task 196
parsl.dataflow.dflow DEBUG: Task 196 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/1f6fa4f7-f087-4c34-9115-360ea4edea75_isrForCrosstalkSources_898387_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/1f6fa4f7-f087-4c34-9115-360ea4edea75_isrForCrosstalkSources_898387_38.stderr
parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34734cee5fd2bdd6a2f401f504408840 with result from task 195
parsl.dataflow.dflow DEBUG: Task 195 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/923197e3-4d32-4efa-aef9-de7736d8f41d_isrForCrosstalkSources_898385_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/923197e3-4d32-4efa-aef9-de7736d8f41d_isrForCrosstalkSources_898385_15.stderr
parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35059e54ec43109cb6f54e273f6544da with result from task 197
parsl.dataflow.dflow DEBUG: Task 197 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/26bfa0aa-3550-4b78-aba1-afbfc3e7a052_isrForCrosstalkSources_898280_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/26bfa0aa-3550-4b78-aba1-afbfc3e7a052_isrForCrosstalkSources_898280_46.stderr
parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c62ae904dfa3db90efa086afd1500283 with result from task 194
parsl.dataflow.dflow DEBUG: Task 194 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0d6d7d4f-da03-45f6-ab98-9535b5315c5c_isrForCrosstalkSources_898386_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0d6d7d4f-da03-45f6-ab98-9535b5315c5c_isrForCrosstalkSources_898386_51.stderr
parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b313740c46ee2081d7606ceb01f1ec57 with result from task 198
parsl.dataflow.dflow DEBUG: Task 198 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/94f62a8b-de51-4436-8290-d60b0408ca4d_isrForCrosstalkSources_898385_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/94f62a8b-de51-4436-8290-d60b0408ca4d_isrForCrosstalkSources_898385_55.stderr
parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7143f7ac3104f3f86a4ff497ccea382f with result from task 192
parsl.dataflow.dflow DEBUG: Task 192 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/a32810c9-d97f-467e-9dd7-ae0b12e5d7f3_isrForCrosstalkSources_898386_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/a32810c9-d97f-467e-9dd7-ae0b12e5d7f3_isrForCrosstalkSources_898386_24.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:50 proc_lsst.multi:146 [INFO]  found job 17001231 in provider checkpoint
proc_lsst.multi INFO: found job 17001231 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:50 proc_lsst.multi:146 [INFO]  found job 17001232 in provider checkpoint
proc_lsst.multi INFO: found job 17001232 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:50 proc_lsst.multi:146 [INFO]  found job 17001233 in provider checkpoint
proc_lsst.multi INFO: found job 17001233 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:51 proc_lsst.multi:146 [INFO]  found job 17001234 in provider checkpoint
proc_lsst.multi INFO: found job 17001234 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:51 proc_lsst.multi:146 [INFO]  found job 17001235 in provider checkpoint
proc_lsst.multi INFO: found job 17001235 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:51 proc_lsst.multi:146 [INFO]  found job 17001236 in provider checkpoint
proc_lsst.multi INFO: found job 17001236 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:51 proc_lsst.multi:146 [INFO]  found job 17001237 in provider checkpoint
proc_lsst.multi INFO: found job 17001237 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:51 proc_lsst.multi:146 [INFO]  found job 17001238 in provider checkpoint
proc_lsst.multi INFO: found job 17001238 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:51 proc_lsst.multi:146 [INFO]  found job 17001240 in provider checkpoint
proc_lsst.multi INFO: found job 17001240 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:52 proc_lsst.multi:146 [INFO]  found job 17001241 in provider checkpoint
proc_lsst.multi INFO: found job 17001241 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:52 proc_lsst.multi:146 [INFO]  found job 17001242 in provider checkpoint
proc_lsst.multi INFO: found job 17001242 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:52 proc_lsst.multi:146 [INFO]  found job 17001243 in provider checkpoint
proc_lsst.multi INFO: found job 17001243 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:52 proc_lsst.multi:146 [INFO]  found job 17001244 in provider checkpoint
proc_lsst.multi INFO: found job 17001244 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:52 proc_lsst.multi:146 [INFO]  found job 17001245 in provider checkpoint
proc_lsst.multi INFO: found job 17001245 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:52 proc_lsst.multi:146 [INFO]  found job 17001246 in provider checkpoint
proc_lsst.multi INFO: found job 17001246 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:53 proc_lsst.multi:146 [INFO]  found job 17001247 in provider checkpoint
proc_lsst.multi INFO: found job 17001247 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f955cf3e6e507686edeb8f3668d98a9 with result from task 212
parsl.dataflow.dflow DEBUG: Task 212 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b83cecca-92d1-4d87-992d-771e4bf14809_isrForCrosstalkSources_898280_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b83cecca-92d1-4d87-992d-771e4bf14809_isrForCrosstalkSources_898280_11.stderr
parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a5e055c674c2655ede94bc097bc1560 with result from task 215
parsl.dataflow.dflow DEBUG: Task 215 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4137ffe0-2a8f-48c8-94cf-32b231226632_isrForCrosstalkSources_898281_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/4137ffe0-2a8f-48c8-94cf-32b231226632_isrForCrosstalkSources_898281_18.stderr
parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c7dff3a44ceeb00b5758138e1206e39a with result from task 208
parsl.dataflow.dflow DEBUG: Task 208 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7d1f7bba-b3cc-4bd4-ba1e-88c3c131bd8c_isrForCrosstalkSources_898385_62.stdout
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/7d1f7bba-b3cc-4bd4-ba1e-88c3c131bd8c_isrForCrosstalkSources_898385_62.stderr
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done)
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e2705bcb174c171effa584eed57f463 with result from task 210
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow DEBUG: Task 210 checkpointed
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/da95cc7e-ab7f-4b6b-901e-1044c46550cf_isrForCrosstalkSources_898385_1.stdout
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/da95cc7e-ab7f-4b6b-901e-1044c46550cf_isrForCrosstalkSources_898385_1.stderr
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done)
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d25d18079d9bc9ccefa14fda09d4c23 with result from task 211
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:53 proc_lsst.multi:146 [INFO]  found job 17001248 in provider checkpoint
parsl.dataflow.dflow DEBUG: Task 211 checkpointed
proc_lsst.multi INFO: found job 17001248 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/df56b7fd-745a-4180-808d-b5ac5829402e_isrForCrosstalkSources_898387_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/df56b7fd-745a-4180-808d-b5ac5829402e_isrForCrosstalkSources_898387_55.stderr
parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1b346318a3bed085baec58135b5f2b8e with result from task 209
parsl.dataflow.dflow DEBUG: Task 209 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ed51c2a9-4323-40b0-81a8-76a79b21269a_isrForCrosstalkSources_898387_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ed51c2a9-4323-40b0-81a8-76a79b21269a_isrForCrosstalkSources_898387_45.stderr
parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 749ff4b1738c2b01cd5825fa97e8a8fe with result from task 214
parsl.dataflow.dflow DEBUG: Task 214 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d4e5a1e1-79a1-462c-bf70-c1669ad412e0_isrForCrosstalkSources_898387_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d4e5a1e1-79a1-462c-bf70-c1669ad412e0_isrForCrosstalkSources_898387_15.stderr
parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 769c87018074adb16b9f8ff09eeae225 with result from task 213
parsl.dataflow.dflow DEBUG: Task 213 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/5e4c32b1-ec94-40d5-9231-67c70f112daf_isrForCrosstalkSources_898282_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/5e4c32b1-ec94-40d5-9231-67c70f112daf_isrForCrosstalkSources_898282_57.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
2024-03-11 23:02:53 proc_lsst.multi:146 [INFO]  found job 17001249 in provider checkpoint
proc_lsst.multi INFO: found job 17001249 in provider checkpoint
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17001212,17001213,17001214,17001215,17001216,17001217,17001218,17001219,17001220,17001221,17001222,17001223,17001224,17001225,17001226,17001227,17001228,17001229,17001230,17001231,17001232,17001233,17001234,17001235,17001236,17001237,17001238,17001240,17001241,17001242,17001243,17001244,17001245,17001246,17001247,17001248,17001249'
parsl.providers.slurm.slurm DEBUG: squeue returned 17001232 R
17001233 R
17001234 R
17001235 R
17001236 R
17001237 R
17001238 R
17001240 R
17001241 R
17001242 R
17001243 R
17001244 R
17001245 R
17001246 R
17001247 R
17001248 R
17001249 R
17001212 R
17001213 R
17001214 R
17001215 R
17001216 R
17001217 R
17001218 R
17001219 R
17001220 R
17001221 R
17001222 R
17001223 R
17001224 R
17001225 R
17001226 R
17001227 R
17001228 R
17001229 R
17001230 R
17001231 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17001232 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001233 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001234 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001235 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001236 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001237 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001238 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001240 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001241 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001242 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001243 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001244 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001245 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001246 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001247 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001248 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001249 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001212 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001213 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001214 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001215 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001216 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001217 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001218 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001219 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001220 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001221 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001222 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001223 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001224 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001225 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001226 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001227 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001228 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001229 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001230 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17001231 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 104
parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 38/55 running/pending blocks, and 298 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 104
parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 38/55 running/pending blocks, and 298 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8588646e2344b7424d71f00665a95c40 with result from task 185
parsl.dataflow.dflow DEBUG: Task 185 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/bdf9e73b-fa9b-471a-984d-2267cf5b8ae1_isrForCrosstalkSources_898386_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/bdf9e73b-fa9b-471a-984d-2267cf5b8ae1_isrForCrosstalkSources_898386_19.stderr
parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5f51f5724e9a210ee74759dbc453d7e6 with result from task 190
parsl.dataflow.dflow DEBUG: Task 190 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/e91ffc52-338e-4338-8fd8-96835e4e6a0b_isrForCrosstalkSources_898282_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/e91ffc52-338e-4338-8fd8-96835e4e6a0b_isrForCrosstalkSources_898282_20.stderr
parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cce1b182ef42eadcadf7324403aae849 with result from task 189
parsl.dataflow.dflow DEBUG: Task 189 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/00ebeb8e-0850-45b2-9ce2-a1b99bcc9672_isrForCrosstalkSources_898282_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/00ebeb8e-0850-45b2-9ce2-a1b99bcc9672_isrForCrosstalkSources_898282_25.stderr
parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c4dc4f05ad1e929c14ead4fa9c578b6c with result from task 183
parsl.dataflow.dflow DEBUG: Task 183 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e175c2e5-b485-4443-866d-2c3c199ae18e_isrForCrosstalkSources_898281_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e175c2e5-b485-4443-866d-2c3c199ae18e_isrForCrosstalkSources_898281_46.stderr
parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7deb2ae1db691f7e3d8fff933bc40b75 with result from task 188
parsl.dataflow.dflow DEBUG: Task 188 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0e36c2e2-a32e-4f5b-8782-4a817a33c912_isrForCrosstalkSources_898386_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0e36c2e2-a32e-4f5b-8782-4a817a33c912_isrForCrosstalkSources_898386_36.stderr
parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35efda2a7955b53e3d4c5d3982451552 with result from task 186
parsl.dataflow.dflow DEBUG: Task 186 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/4a142d0b-6f45-4198-8977-93fc40cf470f_isrForCrosstalkSources_898385_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/4a142d0b-6f45-4198-8977-93fc40cf470f_isrForCrosstalkSources_898385_43.stderr
parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 627f950c2455264809e469a3f300a21a with result from task 187
parsl.dataflow.dflow DEBUG: Task 187 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0233d079-e93b-410e-9b8b-d5aedf7d1b72_isrForCrosstalkSources_898386_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/0233d079-e93b-410e-9b8b-d5aedf7d1b72_isrForCrosstalkSources_898386_41.stderr
parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e8bf6e39b52b4efeacd856f4eba3cb56 with result from task 184
parsl.dataflow.dflow DEBUG: Task 184 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/adcb97b7-fada-460c-90f6-949785ce6356_isrForCrosstalkSources_898280_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/adcb97b7-fada-460c-90f6-949785ce6356_isrForCrosstalkSources_898280_49.stderr
parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eef486e78931f905048dff572bfb0ada with result from task 216
parsl.dataflow.dflow DEBUG: Task 216 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/7cf81f20-46b4-4868-888a-a6bed80f5927_isrForCrosstalkSources_898282_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/7cf81f20-46b4-4868-888a-a6bed80f5927_isrForCrosstalkSources_898282_28.stderr
parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 12c81434deedf95d4493b8a4c1c36c4f with result from task 219
parsl.dataflow.dflow DEBUG: Task 219 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f13024d1-5ca9-4f2f-bb54-697258f87adb_isrForCrosstalkSources_898386_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/f13024d1-5ca9-4f2f-bb54-697258f87adb_isrForCrosstalkSources_898386_40.stderr
parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19f303d1f78cbf297803bbec942db52b with result from task 217
parsl.dataflow.dflow DEBUG: Task 217 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/16997706-3b1c-4ad6-9c12-cda7eb9bd5d9_isrForCrosstalkSources_898387_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/16997706-3b1c-4ad6-9c12-cda7eb9bd5d9_isrForCrosstalkSources_898387_61.stderr
parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6feba2ef27e2057e154d08d523d90df4 with result from task 223
parsl.dataflow.dflow DEBUG: Task 223 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/5019a8c2-2289-4139-89da-c2b2155b72c6_isrForCrosstalkSources_898280_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/5019a8c2-2289-4139-89da-c2b2155b72c6_isrForCrosstalkSources_898280_60.stderr
parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 874ea952fbde3d52b6f4ca5cf5b96f35 with result from task 220
parsl.dataflow.dflow DEBUG: Task 220 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/8d1f44f1-62ae-4e3e-b3d3-0f8cc488da7d_isrForCrosstalkSources_898282_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/8d1f44f1-62ae-4e3e-b3d3-0f8cc488da7d_isrForCrosstalkSources_898282_30.stderr
parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 45fe79ad83c8178b990ea66fd6e6b6a9 with result from task 221
parsl.dataflow.dflow DEBUG: Task 221 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e69ed1de-f30b-4fcc-9b72-096f11075e00_isrForCrosstalkSources_898280_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/e69ed1de-f30b-4fcc-9b72-096f11075e00_isrForCrosstalkSources_898280_39.stderr
parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 57d988e4a648094d04814ad04ea0d1bc with result from task 218
parsl.dataflow.dflow DEBUG: Task 218 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/916f3315-11f9-4040-b09c-442e4e869a6b_isrForCrosstalkSources_898282_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/916f3315-11f9-4040-b09c-442e4e869a6b_isrForCrosstalkSources_898282_26.stderr
parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67e05a211f6208ca7b54730248ad9527 with result from task 222
parsl.dataflow.dflow DEBUG: Task 222 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/be32eff1-d676-4524-a200-50d58670734e_isrForCrosstalkSources_898387_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/be32eff1-d676-4524-a200-50d58670734e_isrForCrosstalkSources_898387_6.stderr
parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 996f5e09b7e1e35449f5f761ef6345c7 with result from task 288
parsl.dataflow.dflow DEBUG: Task 288 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ce912925-2b06-4bff-bd0c-48682412c5e3_isrForCrosstalkSources_898387_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ce912925-2b06-4bff-bd0c-48682412c5e3_isrForCrosstalkSources_898387_17.stderr
parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8dd6da188df676f5bb53f4b27f02d668 with result from task 287
parsl.dataflow.dflow DEBUG: Task 287 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/7b3ef90f-0801-45ac-8357-e8ba9e21a07d_isrForCrosstalkSources_898386_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/7b3ef90f-0801-45ac-8357-e8ba9e21a07d_isrForCrosstalkSources_898386_39.stderr
parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 73c5a29ffd621ab701ebabae4edcb028 with result from task 284
parsl.dataflow.dflow DEBUG: Task 284 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/cc18e4d9-45bd-4666-b7bc-915177711a8a_isrForCrosstalkSources_898282_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/cc18e4d9-45bd-4666-b7bc-915177711a8a_isrForCrosstalkSources_898282_34.stderr
parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry edb7dc15e4464da2b17f9556760f74ab with result from task 252
parsl.dataflow.dflow DEBUG: Task 252 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/13c3129a-4aa8-41f5-8c02-6d4ece4c6d86_isrForCrosstalkSources_898282_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/13c3129a-4aa8-41f5-8c02-6d4ece4c6d86_isrForCrosstalkSources_898282_13.stderr
parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e3985acbce9aba00323d97f99927313 with result from task 248
parsl.dataflow.dflow DEBUG: Task 248 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/45399a4f-e60a-4100-8e36-814ec5e52d16_isrForCrosstalkSources_898281_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/45399a4f-e60a-4100-8e36-814ec5e52d16_isrForCrosstalkSources_898281_27.stderr
parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 954f4e7e1820732168879fd3156c0e8f with result from task 286
parsl.dataflow.dflow DEBUG: Task 286 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ad76c485-a2f9-46b7-9eb2-5fca09128815_isrForCrosstalkSources_898387_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/ad76c485-a2f9-46b7-9eb2-5fca09128815_isrForCrosstalkSources_898387_39.stderr
parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a7c80e865e56ea7d8e7b7b0f41d41ec4 with result from task 283
parsl.dataflow.dflow DEBUG: Task 283 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/b7019ecb-c53e-45f5-91e3-0de71317fe17_isrForCrosstalkSources_898387_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/b7019ecb-c53e-45f5-91e3-0de71317fe17_isrForCrosstalkSources_898387_24.stderr
parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6bb08490d142c92fad18c527ef7f3a3 with result from task 290
parsl.dataflow.dflow DEBUG: Task 290 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/bcb09e7d-14af-421d-a018-b89e4d3154fa_isrForCrosstalkSources_898280_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/bcb09e7d-14af-421d-a018-b89e4d3154fa_isrForCrosstalkSources_898280_47.stderr
parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6005b3614cdba985bdbbc7852abf54a with result from task 289
parsl.dataflow.dflow DEBUG: Task 289 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/db27546c-2e84-4157-ba38-811122d8fb21_isrForCrosstalkSources_898387_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/db27546c-2e84-4157-ba38-811122d8fb21_isrForCrosstalkSources_898387_11.stderr
parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8232299b19256d73d30775b27b7c98a7 with result from task 285
parsl.dataflow.dflow DEBUG: Task 285 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e5db69ad-dffd-4308-afdf-f7aebbb09d76_isrForCrosstalkSources_898385_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e5db69ad-dffd-4308-afdf-f7aebbb09d76_isrForCrosstalkSources_898385_38.stderr
parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2d303c848e6ddf0ce3540d3864176650 with result from task 237
parsl.dataflow.dflow DEBUG: Task 237 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/3cff9b60-d63c-4b4b-824d-8973fbd7528b_isrForCrosstalkSources_898280_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/3cff9b60-d63c-4b4b-824d-8973fbd7528b_isrForCrosstalkSources_898280_26.stderr
parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 85f4ac843a29bd3b427afd1902861aed with result from task 238
parsl.dataflow.dflow DEBUG: Task 238 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e98eb2e4-0b79-4bcd-b972-2a46ee7c1653_isrForCrosstalkSources_898281_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e98eb2e4-0b79-4bcd-b972-2a46ee7c1653_isrForCrosstalkSources_898281_19.stderr
parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a38d286654599146c5164ab7672a4b6b with result from task 239
parsl.dataflow.dflow DEBUG: Task 239 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/57032478-6179-40c1-a7f1-80d5f44bfed9_isrForCrosstalkSources_898282_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/57032478-6179-40c1-a7f1-80d5f44bfed9_isrForCrosstalkSources_898282_5.stderr
parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd2c57e5e1668c63ddf1d0517450e4f2 with result from task 234
parsl.dataflow.dflow DEBUG: Task 234 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/663bb20b-86b1-4dc1-84e3-05dbe2387ebe_isrForCrosstalkSources_898387_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/663bb20b-86b1-4dc1-84e3-05dbe2387ebe_isrForCrosstalkSources_898387_25.stderr
parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53e7c423c09dbb50690dc0495385b2bb with result from task 232
parsl.dataflow.dflow DEBUG: Task 232 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b075fd2c-73db-4d85-a453-96bc1836e085_isrForCrosstalkSources_898282_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/b075fd2c-73db-4d85-a453-96bc1836e085_isrForCrosstalkSources_898282_14.stderr
parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cebe02c893df9bf483e5814b6fdc1817 with result from task 250
parsl.dataflow.dflow DEBUG: Task 250 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/fd427fd4-c3f1-46c0-a981-14f079528eeb_isrForCrosstalkSources_898386_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/fd427fd4-c3f1-46c0-a981-14f079528eeb_isrForCrosstalkSources_898386_50.stderr
parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9db114d0c04316f0332efe25d4a0596f with result from task 253
parsl.dataflow.dflow DEBUG: Task 253 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/6d8317f5-9faf-4eaf-a708-f53def25bfa2_isrForCrosstalkSources_898282_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/6d8317f5-9faf-4eaf-a708-f53def25bfa2_isrForCrosstalkSources_898282_43.stderr
parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed5a1b69e11b015c837b53f04fb13c51 with result from task 255
parsl.dataflow.dflow DEBUG: Task 255 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/25624f14-5d18-4fc0-93f1-eaa34ddd3f2d_isrForCrosstalkSources_898385_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/25624f14-5d18-4fc0-93f1-eaa34ddd3f2d_isrForCrosstalkSources_898385_2.stderr
parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28ece017ad356b66eae687baa2695163 with result from task 254
parsl.dataflow.dflow DEBUG: Task 254 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bd83d7e2-017c-4901-93dd-e239ee4604ff_isrForCrosstalkSources_898282_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/bd83d7e2-017c-4901-93dd-e239ee4604ff_isrForCrosstalkSources_898282_53.stderr
parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91b59da9cb3778693119176ac8b622b3 with result from task 249
parsl.dataflow.dflow DEBUG: Task 249 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bac37693-60dd-456e-bf2e-8a24de2ee18b_isrForCrosstalkSources_898281_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/bac37693-60dd-456e-bf2e-8a24de2ee18b_isrForCrosstalkSources_898281_4.stderr
parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9ee17cadb7fc33840499c971b92688c with result from task 251
parsl.dataflow.dflow DEBUG: Task 251 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/75333039-d099-413d-8796-38a9baaefb29_isrForCrosstalkSources_898386_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/75333039-d099-413d-8796-38a9baaefb29_isrForCrosstalkSources_898386_43.stderr
parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f90bb08e73d1a2a833811b296e060521 with result from task 291
parsl.dataflow.dflow DEBUG: Task 291 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/2c3e2fa6-d66a-44d8-a268-cdff0f67bf4b_isrForCrosstalkSources_898385_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/2c3e2fa6-d66a-44d8-a268-cdff0f67bf4b_isrForCrosstalkSources_898385_25.stderr
parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0195518e8936a32ec2110a684cd6d523 with result from task 292
parsl.dataflow.dflow DEBUG: Task 292 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c2ac3626-c307-4e7b-a9dc-c3649a62420e_isrForCrosstalkSources_898282_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/c2ac3626-c307-4e7b-a9dc-c3649a62420e_isrForCrosstalkSources_898282_35.stderr
parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62385d171b99f6ca57b6138c2f874499 with result from task 298
parsl.dataflow.dflow DEBUG: Task 298 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2355ec33-bf8b-4f1d-ad59-3b5b3f73254f_isrForCrosstalkSources_898281_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2355ec33-bf8b-4f1d-ad59-3b5b3f73254f_isrForCrosstalkSources_898281_58.stderr
parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47a10c478780851812436f1f1facb9e2 with result from task 295
parsl.dataflow.dflow DEBUG: Task 295 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ef421e5c-8c8d-443b-8ece-9e1d87006410_isrForCrosstalkSources_898280_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/ef421e5c-8c8d-443b-8ece-9e1d87006410_isrForCrosstalkSources_898280_17.stderr
parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed6fea2e46724c738dbfafde6fc65f2f with result from task 293
parsl.dataflow.dflow DEBUG: Task 293 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/6c098e60-1588-4db7-bbaa-666abd9f184b_isrForCrosstalkSources_898280_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/6c098e60-1588-4db7-bbaa-666abd9f184b_isrForCrosstalkSources_898280_37.stderr
parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 150a3652f9748d73a5d1136575c5c129 with result from task 235
parsl.dataflow.dflow DEBUG: Task 235 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4588621c-fefb-4ba5-a0c7-49580bf8d7ab_isrForCrosstalkSources_898282_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/4588621c-fefb-4ba5-a0c7-49580bf8d7ab_isrForCrosstalkSources_898282_59.stderr
parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dbc93e0640874db9282e11e89d34b192 with result from task 233
parsl.dataflow.dflow DEBUG: Task 233 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/f27d660e-c7e2-4c87-ad77-f174158bf067_isrForCrosstalkSources_898282_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/f27d660e-c7e2-4c87-ad77-f174158bf067_isrForCrosstalkSources_898282_2.stderr
parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4e081cf1b700092b7b3d810b3000919d with result from task 297
parsl.dataflow.dflow DEBUG: Task 297 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/ebf0da5e-d4f5-4348-82d2-91fbc129d3b6_isrForCrosstalkSources_898281_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/ebf0da5e-d4f5-4348-82d2-91fbc129d3b6_isrForCrosstalkSources_898281_21.stderr
parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c9a50e24bdc27092ad1e453e429bdf7 with result from task 294
parsl.dataflow.dflow DEBUG: Task 294 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e226cec5-918c-40f7-81b3-8a5372400631_isrForCrosstalkSources_898281_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/e226cec5-918c-40f7-81b3-8a5372400631_isrForCrosstalkSources_898281_54.stderr
parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 723fba16a9b22a4e0160551aff5e8c22 with result from task 296
parsl.dataflow.dflow DEBUG: Task 296 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/81f6a94d-5235-4a76-b349-77f970520259_isrForCrosstalkSources_898386_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/81f6a94d-5235-4a76-b349-77f970520259_isrForCrosstalkSources_898386_52.stderr
parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36f9252e96f69866a35cebae61fe572d with result from task 236
parsl.dataflow.dflow DEBUG: Task 236 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/57ca9658-be51-4b85-b760-71709da9e0b2_isrForCrosstalkSources_898281_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/57ca9658-be51-4b85-b760-71709da9e0b2_isrForCrosstalkSources_898281_45.stderr
parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 68f0865dad9b690b3998019f125a925f with result from task 228
parsl.dataflow.dflow DEBUG: Task 228 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d9908ed3-02b9-4b0a-be4f-caac7cc7043b_isrForCrosstalkSources_898282_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d9908ed3-02b9-4b0a-be4f-caac7cc7043b_isrForCrosstalkSources_898282_19.stderr
parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88cbdefd2c08cb3e8e5962e1a6c5175a with result from task 227
parsl.dataflow.dflow DEBUG: Task 227 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7f2222d3-ed87-4e48-a449-b742de417a56_isrForCrosstalkSources_898281_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/7f2222d3-ed87-4e48-a449-b742de417a56_isrForCrosstalkSources_898281_24.stderr
parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3ca77446260ced1f19351ebc34dfd104 with result from task 231
parsl.dataflow.dflow DEBUG: Task 231 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/c62ab383-1519-44c2-bcb0-707d43d30e94_isrForCrosstalkSources_898281_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/c62ab383-1519-44c2-bcb0-707d43d30e94_isrForCrosstalkSources_898281_17.stderr
parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fc5d9c3defd10b1dbc79c163c969949a with result from task 225
parsl.dataflow.dflow DEBUG: Task 225 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2fd686ce-a8e3-4b53-aa3e-2dd3bd3431ff_isrForCrosstalkSources_898386_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/2fd686ce-a8e3-4b53-aa3e-2dd3bd3431ff_isrForCrosstalkSources_898386_12.stderr
parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e8f507270ad4764ae6ed236f8d2eaa1 with result from task 226
parsl.dataflow.dflow DEBUG: Task 226 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/19605d16-05e9-4e1c-aff9-6afe3873ebc2_isrForCrosstalkSources_898280_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/19605d16-05e9-4e1c-aff9-6afe3873ebc2_isrForCrosstalkSources_898280_33.stderr
parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 98e92fdb1cc3f363816ba8a928e52e59 with result from task 224
parsl.dataflow.dflow DEBUG: Task 224 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b8949b8c-0a3b-4e1c-b30c-82a214de2394_isrForCrosstalkSources_898280_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/b8949b8c-0a3b-4e1c-b30c-82a214de2394_isrForCrosstalkSources_898280_30.stderr
parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4c5eb7c59518dc55a47e114ddfd62aad with result from task 259
parsl.dataflow.dflow DEBUG: Task 259 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/dbfd9b4c-27d7-4496-a29f-1bff15ba286b_isrForCrosstalkSources_898386_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/dbfd9b4c-27d7-4496-a29f-1bff15ba286b_isrForCrosstalkSources_898386_54.stderr
parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aabbf9c40104759a0bb140c4824f99c7 with result from task 262
parsl.dataflow.dflow DEBUG: Task 262 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d3bd7965-655e-4993-9da4-62bf7985a08e_isrForCrosstalkSources_898387_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d3bd7965-655e-4993-9da4-62bf7985a08e_isrForCrosstalkSources_898387_16.stderr
parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f146db9415a7100fcb25f673da6e2044 with result from task 244
parsl.dataflow.dflow DEBUG: Task 244 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/6c93c632-f7f1-4e01-9566-4de2a0132ce0_isrForCrosstalkSources_898280_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/6c93c632-f7f1-4e01-9566-4de2a0132ce0_isrForCrosstalkSources_898280_51.stderr
parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5524f6c6cf8d3774e776ddae56ea9ee0 with result from task 247
parsl.dataflow.dflow DEBUG: Task 247 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6e983895-8d1d-4efc-95b9-25ca103bcd29_isrForCrosstalkSources_898385_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/6e983895-8d1d-4efc-95b9-25ca103bcd29_isrForCrosstalkSources_898385_16.stderr
parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e6ab2f443879038aaba0a85bc961551 with result from task 242
parsl.dataflow.dflow DEBUG: Task 242 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/40d56106-ebc6-4b6a-a4fa-ee5703c5f99c_isrForCrosstalkSources_898281_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/40d56106-ebc6-4b6a-a4fa-ee5703c5f99c_isrForCrosstalkSources_898281_36.stderr
parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db67900bb9fbc356940f35c3e225a5be with result from task 243
parsl.dataflow.dflow DEBUG: Task 243 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/fbf05147-d949-44ad-9904-f2ee06299757_isrForCrosstalkSources_898280_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/fbf05147-d949-44ad-9904-f2ee06299757_isrForCrosstalkSources_898280_45.stderr
parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 629a08d4e76d26278618826f569ec2b8 with result from task 264
parsl.dataflow.dflow DEBUG: Task 264 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e1db5d57-058d-4dca-927e-5486c03cded6_isrForCrosstalkSources_898385_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/e1db5d57-058d-4dca-927e-5486c03cded6_isrForCrosstalkSources_898385_37.stderr
parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a8ebb6615f442e52d6b9d60ac8acfcfd with result from task 260
parsl.dataflow.dflow DEBUG: Task 260 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/26381f99-274e-4da1-937b-0fca039f42e3_isrForCrosstalkSources_898387_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/26381f99-274e-4da1-937b-0fca039f42e3_isrForCrosstalkSources_898387_62.stderr
parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a737065167aaf209a7f8fc65dd669836 with result from task 266
parsl.dataflow.dflow DEBUG: Task 266 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2136e3e6-f57b-4c94-baeb-3bd35c2c18c8_isrForCrosstalkSources_898281_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/2136e3e6-f57b-4c94-baeb-3bd35c2c18c8_isrForCrosstalkSources_898281_47.stderr
parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f01dce1690bbdf50b0ff3ef1bb5751ce with result from task 261
parsl.dataflow.dflow DEBUG: Task 261 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/097d3af5-2f19-4340-ab7b-e87d8f9e2b57_isrForCrosstalkSources_898386_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/097d3af5-2f19-4340-ab7b-e87d8f9e2b57_isrForCrosstalkSources_898386_47.stderr
parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36cc30de8cc31d5351ff2361999b0d74 with result from task 263
parsl.dataflow.dflow DEBUG: Task 263 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/01bd5c5d-c125-451f-8711-ce6da8560465_isrForCrosstalkSources_898280_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/01bd5c5d-c125-451f-8711-ce6da8560465_isrForCrosstalkSources_898280_3.stderr
parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c080fca29df9f8befef7b878afdc2737 with result from task 265
parsl.dataflow.dflow DEBUG: Task 265 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/ad232190-ed85-43fe-82f9-1b991387cd48_isrForCrosstalkSources_898386_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/ad232190-ed85-43fe-82f9-1b991387cd48_isrForCrosstalkSources_898386_53.stderr
parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4dd5db412a071d3993006d4c7cfba9d6 with result from task 241
parsl.dataflow.dflow DEBUG: Task 241 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/aa9c5689-65c5-4930-995a-0e2837e8b351_isrForCrosstalkSources_898280_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/aa9c5689-65c5-4930-995a-0e2837e8b351_isrForCrosstalkSources_898280_8.stderr
parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5eb724d0989e0573ac68e02473aed837 with result from task 240
parsl.dataflow.dflow DEBUG: Task 240 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/732a4187-75c1-4773-b469-71c117fef941_isrForCrosstalkSources_898281_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/732a4187-75c1-4773-b469-71c117fef941_isrForCrosstalkSources_898281_48.stderr
parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8e0d6c60deda890ed21d068ee939fe6 with result from task 246
parsl.dataflow.dflow DEBUG: Task 246 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/62fb6b8b-829e-4169-85f4-06dcfcd4d9c3_isrForCrosstalkSources_898386_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/62fb6b8b-829e-4169-85f4-06dcfcd4d9c3_isrForCrosstalkSources_898386_42.stderr
parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6bc44e9b5e2d65274d38a6237808ccb5 with result from task 245
parsl.dataflow.dflow DEBUG: Task 245 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/de2c59f2-6a71-4928-ae14-0eee8c17b3a9_isrForCrosstalkSources_898280_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/de2c59f2-6a71-4928-ae14-0eee8c17b3a9_isrForCrosstalkSources_898280_32.stderr
parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 270b821ec670a2794cecf43c9905d4d6 with result from task 273
parsl.dataflow.dflow DEBUG: Task 273 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/87dfc4f7-0c68-4f58-9ab7-2e95540f72b4_isrForCrosstalkSources_898280_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/87dfc4f7-0c68-4f58-9ab7-2e95540f72b4_isrForCrosstalkSources_898280_44.stderr
parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1265d0b769f6b3851d47015ac6af91fe with result from task 268
parsl.dataflow.dflow DEBUG: Task 268 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/b066b15e-2b20-4d6b-a994-a3062c0af9fd_isrForCrosstalkSources_898281_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/b066b15e-2b20-4d6b-a994-a3062c0af9fd_isrForCrosstalkSources_898281_44.stderr
parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7dbeb46f2a44e768f20e4c07557a9bc7 with result from task 274
parsl.dataflow.dflow DEBUG: Task 274 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/59a6ddc0-41a9-4976-b86f-3203d5ac31d1_isrForCrosstalkSources_898282_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/59a6ddc0-41a9-4976-b86f-3203d5ac31d1_isrForCrosstalkSources_898282_4.stderr
parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 90d3fdfdf1dda6983163121e9fe41c10 with result from task 271
parsl.dataflow.dflow DEBUG: Task 271 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d3b9d9cf-e01a-42fe-aca6-89e4422e8d64_isrForCrosstalkSources_898282_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/d3b9d9cf-e01a-42fe-aca6-89e4422e8d64_isrForCrosstalkSources_898282_45.stderr
parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 70a9315cc61fa358628467c3bd5c8cd2 with result from task 272
parsl.dataflow.dflow DEBUG: Task 272 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/96e341cb-9407-47ff-ae2b-c253577aa575_isrForCrosstalkSources_898281_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/96e341cb-9407-47ff-ae2b-c253577aa575_isrForCrosstalkSources_898281_40.stderr
parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 555cd136355fdff67b6a241c6b315f67 with result from task 270
parsl.dataflow.dflow DEBUG: Task 270 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/52a8abee-fa66-43f2-9e52-6861d0de45df_isrForCrosstalkSources_898386_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/52a8abee-fa66-43f2-9e52-6861d0de45df_isrForCrosstalkSources_898386_44.stderr
parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d69769a7f711ee79b83be96235da1e62 with result from task 269
parsl.dataflow.dflow DEBUG: Task 269 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/8fbc3b9d-263b-4d2a-a37c-a942f5280c0d_isrForCrosstalkSources_898385_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/8fbc3b9d-263b-4d2a-a37c-a942f5280c0d_isrForCrosstalkSources_898385_12.stderr
parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 138be5193ca1679ad28ea1a9c0b17e75 with result from task 267
parsl.dataflow.dflow DEBUG: Task 267 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2330cafa-715b-44e2-b9f9-ddd501f4ce5b_isrForCrosstalkSources_898282_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/2330cafa-715b-44e2-b9f9-ddd501f4ce5b_isrForCrosstalkSources_898282_49.stderr
parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 06281ffcc4c68cb96565492ae1823377 with result from task 230
parsl.dataflow.dflow DEBUG: Task 230 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/58be7d9c-0b49-434c-a36b-0ce758c4125c_isrForCrosstalkSources_898386_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/58be7d9c-0b49-434c-a36b-0ce758c4125c_isrForCrosstalkSources_898386_49.stderr
parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 067e572cd996e4cb5dd63bf63b2b90a9 with result from task 229
parsl.dataflow.dflow DEBUG: Task 229 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/eb6440f8-06fe-4ec1-93b5-68d8e8448682_isrForCrosstalkSources_898281_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/eb6440f8-06fe-4ec1-93b5-68d8e8448682_isrForCrosstalkSources_898281_1.stderr
parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d5437b5fc2d40215f4524850d7a5a4a with result from task 306
parsl.dataflow.dflow DEBUG: Task 306 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a5f72ee0-fe0c-4d12-8095-b7accdd905ca_isrForCrosstalkSources_898385_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a5f72ee0-fe0c-4d12-8095-b7accdd905ca_isrForCrosstalkSources_898385_7.stderr
parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f17eb644fe17b8458419d3ed4b3d0669 with result from task 303
parsl.dataflow.dflow DEBUG: Task 303 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/a105e266-a9d5-43f2-beb7-59fcd5a7a6b2_isrForCrosstalkSources_898387_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/a105e266-a9d5-43f2-beb7-59fcd5a7a6b2_isrForCrosstalkSources_898387_5.stderr
parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8c43b4db44ab22de8de7b96ab39b4d62 with result from task 305
parsl.dataflow.dflow DEBUG: Task 305 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/3cef908e-c9f6-4633-8b4b-ed739fc5bcd6_isrForCrosstalkSources_898280_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898280/3cef908e-c9f6-4633-8b4b-ed739fc5bcd6_isrForCrosstalkSources_898280_61.stderr
parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6957b27d2689400bb5b87f430ae0c9f with result from task 299
parsl.dataflow.dflow DEBUG: Task 299 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/db93e938-7210-42ae-b933-5fc6d97590a1_isrForCrosstalkSources_898386_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/db93e938-7210-42ae-b933-5fc6d97590a1_isrForCrosstalkSources_898386_9.stderr
parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8029c3bbb38cbc1f42e680117babb5b with result from task 302
parsl.dataflow.dflow DEBUG: Task 302 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/95dc356a-5ef5-4914-a5d4-dc0188bbcda5_isrForCrosstalkSources_898387_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/95dc356a-5ef5-4914-a5d4-dc0188bbcda5_isrForCrosstalkSources_898387_23.stderr
parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b96834f027a1e3babf2128a2993b960d with result from task 301
parsl.dataflow.dflow DEBUG: Task 301 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/578d3715-5c58-4eab-b24f-f17727e6b8f1_isrForCrosstalkSources_898281_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/578d3715-5c58-4eab-b24f-f17727e6b8f1_isrForCrosstalkSources_898281_61.stderr
parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e518120143357ccc1f68ab48611102c3 with result from task 304
parsl.dataflow.dflow DEBUG: Task 304 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/40d68480-6578-401c-8afb-0714e6c84d7d_isrForCrosstalkSources_898386_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/40d68480-6578-401c-8afb-0714e6c84d7d_isrForCrosstalkSources_898386_38.stderr
parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3592871d5b387151594e3461b065c94a with result from task 300
parsl.dataflow.dflow DEBUG: Task 300 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/362c4606-e4dd-4cde-bcb7-f25619c5e760_isrForCrosstalkSources_898385_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/362c4606-e4dd-4cde-bcb7-f25619c5e760_isrForCrosstalkSources_898385_18.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 16
parsl.dataflow.strategy DEBUG: Executor multi has 16 active tasks, 38/55 running/pending blocks, and 298 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23247697359760
parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4b5cd24dfabca221ef80414adb4d35b4 with result from task 278
parsl.dataflow.dflow DEBUG: Task 278 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a9fb0f38-7430-4bb4-8273-39ba3a470804_isrForCrosstalkSources_898385_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/a9fb0f38-7430-4bb4-8273-39ba3a470804_isrForCrosstalkSources_898385_27.stderr
parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37e09388f7e40f991ae3f8ae2a3dba22 with result from task 277
parsl.dataflow.dflow DEBUG: Task 277 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/c19e8196-d555-40f5-afbc-96b7ca3845ef_isrForCrosstalkSources_898387_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/c19e8196-d555-40f5-afbc-96b7ca3845ef_isrForCrosstalkSources_898387_18.stderr
parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5c83dd016c5f0017203551bd6e3dfed8 with result from task 276
parsl.dataflow.dflow DEBUG: Task 276 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/084b128f-2c05-4e60-bd63-79f281432299_isrForCrosstalkSources_898385_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/084b128f-2c05-4e60-bd63-79f281432299_isrForCrosstalkSources_898385_46.stderr
parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0459c292d75caf3db9643c201c34688 with result from task 275
parsl.dataflow.dflow DEBUG: Task 275 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/0c9ad12a-090d-437d-a152-ae44fc6289ae_isrForCrosstalkSources_898385_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/0c9ad12a-090d-437d-a152-ae44fc6289ae_isrForCrosstalkSources_898385_47.stderr
parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60b2a48e2c0476ac3752a40b8ed0570c with result from task 281
parsl.dataflow.dflow DEBUG: Task 281 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/bd381560-70ff-4804-a4b1-90c89eb46588_isrForCrosstalkSources_898385_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/bd381560-70ff-4804-a4b1-90c89eb46588_isrForCrosstalkSources_898385_5.stderr
parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d97d9ca989717b593a392aa8a378c0b0 with result from task 279
parsl.dataflow.dflow DEBUG: Task 279 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d3ffd0f0-2bf0-48f4-96e7-d2ddcda65dfb_isrForCrosstalkSources_898387_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/d3ffd0f0-2bf0-48f4-96e7-d2ddcda65dfb_isrForCrosstalkSources_898387_47.stderr
parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 178eba76f83e649628e8c4b07aed4e2a with result from task 280
parsl.dataflow.dflow DEBUG: Task 280 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/320e9ff1-381c-4c33-896b-c77d52af6c34_isrForCrosstalkSources_898385_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/320e9ff1-381c-4c33-896b-c77d52af6c34_isrForCrosstalkSources_898385_19.stderr
parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa476bd78f980e2789a86bbe0e4dc337 with result from task 282
parsl.dataflow.dflow DEBUG: Task 282 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/72d7ac71-4984-45f6-820a-d2b6f07d2e1d_isrForCrosstalkSources_898387_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898387/72d7ac71-4984-45f6-820a-d2b6f07d2e1d_isrForCrosstalkSources_898387_37.stderr
parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87a572efbc3b0a9fed04f6ccae29d2ed with result from task 313
parsl.dataflow.dflow DEBUG: Task 313 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/32c19c90-3a57-4f73-8217-0e24a5018307_isrForCrosstalkSources_898385_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/32c19c90-3a57-4f73-8217-0e24a5018307_isrForCrosstalkSources_898385_23.stderr
parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10364d1fa1d23692e27f2b602ebcbab6 with result from task 308
parsl.dataflow.dflow DEBUG: Task 308 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/5ca1e994-e8e3-411b-9492-161cbdd8bd9d_isrForCrosstalkSources_898281_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/5ca1e994-e8e3-411b-9492-161cbdd8bd9d_isrForCrosstalkSources_898281_38.stderr
parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8cba34eab67d28f6af7d136c0f80f25f with result from task 310
parsl.dataflow.dflow DEBUG: Task 310 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/33e9dc29-fe49-4fb7-8d60-2931032d48c9_isrForCrosstalkSources_898282_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/33e9dc29-fe49-4fb7-8d60-2931032d48c9_isrForCrosstalkSources_898282_1.stderr
parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef7e4b4c087b1ef35b9a3a9cf74b9153 with result from task 307
parsl.dataflow.dflow DEBUG: Task 307 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/66d4d493-1b95-4161-aa48-5e2ad5cc8364_isrForCrosstalkSources_898386_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/66d4d493-1b95-4161-aa48-5e2ad5cc8364_isrForCrosstalkSources_898386_28.stderr
parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d418f68a373f4662f492c9aeea6105e8 with result from task 311
parsl.dataflow.dflow DEBUG: Task 311 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/3568f671-d27b-4d32-bf75-7e692220f4ef_isrForCrosstalkSources_898282_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898282/3568f671-d27b-4d32-bf75-7e692220f4ef_isrForCrosstalkSources_898282_7.stderr
parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a094c5dd769a80917a6bd2eea3b5124 with result from task 312
parsl.dataflow.dflow DEBUG: Task 312 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/65fb7dfd-3bee-40d8-93ec-b5ac8aeb07d0_isrForCrosstalkSources_898281_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898281/65fb7dfd-3bee-40d8-93ec-b5ac8aeb07d0_isrForCrosstalkSources_898281_55.stderr
parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f11dd5124965f66b1b02c7718319812 with result from task 314
parsl.dataflow.dflow DEBUG: Task 314 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/ce8c4e42-6dea-4ca4-9d4d-16663820195b_isrForCrosstalkSources_898386_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898386/ce8c4e42-6dea-4ca4-9d4d-16663820195b_isrForCrosstalkSources_898386_58.stderr
parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb2f6eb61e03084d81fafd55b3a3d47b with result from task 309
parsl.dataflow.dflow INFO: DFK cleanup initiated
parsl.dataflow.dflow INFO: Summary of tasks in DFK:
parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0
parsl.dataflow.dflow INFO: Tasks in state States.pending: 0
parsl.dataflow.dflow INFO: Tasks in state States.running: 0
parsl.dataflow.dflow DEBUG: Task 309 checkpointed
parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 372
parsl.dataflow.dflow INFO: Tasks in state States.failed: 0
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/772d1418-66b3-4e80-a04b-da26afd7026e_isrForCrosstalkSources_898385_60.stdout
parsl.dataflow.dflow INFO: Tasks in state States.launched: 0
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/submit/DEEP/20190926/B1b/science#step0/20240311T225937Z/logs/isrForCrosstalkSources/898385/772d1418-66b3-4e80-a04b-da26afd7026e_isrForCrosstalkSources_898385_60.stderr
parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0
parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0
parsl.dataflow.dflow INFO: Tasks in state States.joining: 0
parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0
parsl.dataflow.dflow INFO: End of summary
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Closing job status poller
parsl.dataflow.dflow INFO: Terminated job status poller
parsl.dataflow.dflow INFO: Scaling in and shutting down executors
parsl.dataflow.dflow INFO: Scaling in executor multi
parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[]
parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 38 blocks
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 24680fcc7a57
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 24680fcc7a57
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1732139d2788
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1732139d2788
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9bc25e49bdf3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9bc25e49bdf3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ffe63c8a795d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ffe63c8a795d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7d2f629b63c7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7d2f629b63c7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e8f136a75c69
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e8f136a75c69
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 48d2b730e102
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 48d2b730e102
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a568a09530c9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a568a09530c9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b0da56751e0c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b0da56751e0c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9bf30abf5dcc
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9bf30abf5dcc
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bcc19e0f1500
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bcc19e0f1500
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ef19b75ae008
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ef19b75ae008
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 33a0ae548e82
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 33a0ae548e82
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f93c062df68d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f93c062df68d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 323cb606f470
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 323cb606f470
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 136cdc363123
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 136cdc363123
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7904f48a8526
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7904f48a8526
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 134786ace4e6
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 134786ace4e6
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2e9963e3ed5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2e9963e3ed5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2bd234df695e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2bd234df695e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0e48799d9f38
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0e48799d9f38
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7bdd47790a41
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7bdd47790a41
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7c0f9a9f5868
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7c0f9a9f5868
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e7114fc980eb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e7114fc980eb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 135285b9bd38
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 135285b9bd38
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 47187644ff74
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 47187644ff74
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8a864731dcf
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8a864731dcf
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 40a2098ca9e3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 40a2098ca9e3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe6ad7189609
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe6ad7189609
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d4b6e0979d55
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d4b6e0979d55
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b09e96c72f1e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b09e96c72f1e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e63906d7cc0b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e63906d7cc0b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce683dc38b15
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce683dc38b15
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 197d9080cc00
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 197d9080cc00
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 09ee4a7a2622
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 09ee4a7a2622
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c0d50284aaf7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c0d50284aaf7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 877b73246b6f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 877b73246b6f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 68d31488599d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 68d31488599d
2024-03-11 23:03:01 proc_lsst.multi:146 [INFO]  found job 17001246 in provider checkpoint
proc_lsst.multi INFO: found job 17001246 in provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:201 [INFO]  cancelling 17001246 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001246 on provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:146 [INFO]  found job 17001241 in provider checkpoint
proc_lsst.multi INFO: found job 17001241 in provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:201 [INFO]  cancelling 17001241 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001241 on provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:146 [INFO]  found job 17001245 in provider checkpoint
proc_lsst.multi INFO: found job 17001245 in provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:201 [INFO]  cancelling 17001245 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001245 on provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:146 [INFO]  found job 17001244 in provider checkpoint
proc_lsst.multi INFO: found job 17001244 in provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:201 [INFO]  cancelling 17001244 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001244 on provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:146 [INFO]  found job 17001237 in provider checkpoint
proc_lsst.multi INFO: found job 17001237 in provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:201 [INFO]  cancelling 17001237 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001237 on provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:146 [INFO]  found job 17001236 in provider checkpoint
proc_lsst.multi INFO: found job 17001236 in provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:201 [INFO]  cancelling 17001236 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001236 on provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:146 [INFO]  found job 17001232 in provider checkpoint
proc_lsst.multi INFO: found job 17001232 in provider checkpoint
2024-03-11 23:03:01 proc_lsst.multi:201 [INFO]  cancelling 17001232 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001232 on provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:146 [INFO]  found job 17001234 in provider checkpoint
proc_lsst.multi INFO: found job 17001234 in provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:201 [INFO]  cancelling 17001234 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001234 on provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:146 [INFO]  found job 17001243 in provider checkpoint
proc_lsst.multi INFO: found job 17001243 in provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:201 [INFO]  cancelling 17001243 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001243 on provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:146 [INFO]  found job 17001238 in provider checkpoint
proc_lsst.multi INFO: found job 17001238 in provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:201 [INFO]  cancelling 17001238 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001238 on provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:146 [INFO]  found job 17001240 in provider checkpoint
proc_lsst.multi INFO: found job 17001240 in provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:201 [INFO]  cancelling 17001240 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001240 on provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:146 [INFO]  found job 17001242 in provider checkpoint
proc_lsst.multi INFO: found job 17001242 in provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:201 [INFO]  cancelling 17001242 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001242 on provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:146 [INFO]  found job 17001233 in provider checkpoint
proc_lsst.multi INFO: found job 17001233 in provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:201 [INFO]  cancelling 17001233 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001233 on provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:146 [INFO]  found job 17001248 in provider checkpoint
proc_lsst.multi INFO: found job 17001248 in provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:201 [INFO]  cancelling 17001248 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001248 on provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:146 [INFO]  found job 17001247 in provider checkpoint
proc_lsst.multi INFO: found job 17001247 in provider checkpoint
2024-03-11 23:03:02 proc_lsst.multi:201 [INFO]  cancelling 17001247 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001247 on provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:146 [INFO]  found job 17001249 in provider checkpoint
proc_lsst.multi INFO: found job 17001249 in provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:201 [INFO]  cancelling 17001249 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001249 on provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:146 [INFO]  found job 17001217 in provider checkpoint
proc_lsst.multi INFO: found job 17001217 in provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:201 [INFO]  cancelling 17001217 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001217 on provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:146 [INFO]  found job 17001224 in provider checkpoint
proc_lsst.multi INFO: found job 17001224 in provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:201 [INFO]  cancelling 17001224 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001224 on provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:146 [INFO]  found job 17001218 in provider checkpoint
proc_lsst.multi INFO: found job 17001218 in provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:201 [INFO]  cancelling 17001218 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001218 on provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:146 [INFO]  found job 17001229 in provider checkpoint
proc_lsst.multi INFO: found job 17001229 in provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:201 [INFO]  cancelling 17001229 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001229 on provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:146 [INFO]  found job 17001225 in provider checkpoint
proc_lsst.multi INFO: found job 17001225 in provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:201 [INFO]  cancelling 17001225 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001225 on provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:146 [INFO]  found job 17001228 in provider checkpoint
proc_lsst.multi INFO: found job 17001228 in provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:201 [INFO]  cancelling 17001228 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001228 on provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:146 [INFO]  found job 17001214 in provider checkpoint
proc_lsst.multi INFO: found job 17001214 in provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:201 [INFO]  cancelling 17001214 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001214 on provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:146 [INFO]  found job 17001230 in provider checkpoint
proc_lsst.multi INFO: found job 17001230 in provider checkpoint
2024-03-11 23:03:03 proc_lsst.multi:201 [INFO]  cancelling 17001230 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001230 on provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:146 [INFO]  found job 17001215 in provider checkpoint
proc_lsst.multi INFO: found job 17001215 in provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:201 [INFO]  cancelling 17001215 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001215 on provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:146 [INFO]  found job 17001226 in provider checkpoint
proc_lsst.multi INFO: found job 17001226 in provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:201 [INFO]  cancelling 17001226 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001226 on provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:146 [INFO]  found job 26021 in provider local
proc_lsst.multi INFO: found job 26021 in provider local
2024-03-11 23:03:04 proc_lsst.multi:201 [INFO]  cancelling 26021 on provider local
proc_lsst.multi INFO: cancelling 26021 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 26021
2024-03-11 23:03:04 proc_lsst.multi:146 [INFO]  found job 17001220 in provider checkpoint
proc_lsst.multi INFO: found job 17001220 in provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:201 [INFO]  cancelling 17001220 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001220 on provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:146 [INFO]  found job 17001221 in provider checkpoint
proc_lsst.multi INFO: found job 17001221 in provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:201 [INFO]  cancelling 17001221 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001221 on provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:146 [INFO]  found job 17001213 in provider checkpoint
proc_lsst.multi INFO: found job 17001213 in provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:201 [INFO]  cancelling 17001213 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001213 on provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:146 [INFO]  found job 17001219 in provider checkpoint
proc_lsst.multi INFO: found job 17001219 in provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:201 [INFO]  cancelling 17001219 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001219 on provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:146 [INFO]  found job 17001216 in provider checkpoint
proc_lsst.multi INFO: found job 17001216 in provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:201 [INFO]  cancelling 17001216 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001216 on provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:146 [INFO]  found job 17001212 in provider checkpoint
proc_lsst.multi INFO: found job 17001212 in provider checkpoint
2024-03-11 23:03:04 proc_lsst.multi:201 [INFO]  cancelling 17001212 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001212 on provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:146 [INFO]  found job 17001222 in provider checkpoint
proc_lsst.multi INFO: found job 17001222 in provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:201 [INFO]  cancelling 17001222 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001222 on provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:146 [INFO]  found job 17001227 in provider checkpoint
proc_lsst.multi INFO: found job 17001227 in provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:201 [INFO]  cancelling 17001227 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001227 on provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:146 [INFO]  found job 17001223 in provider checkpoint
proc_lsst.multi INFO: found job 17001223 in provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:201 [INFO]  cancelling 17001223 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001223 on provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:146 [INFO]  found job 17001231 in provider checkpoint
proc_lsst.multi INFO: found job 17001231 in provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:201 [INFO]  cancelling 17001231 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001231 on provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:146 [INFO]  found job 17001235 in provider checkpoint
proc_lsst.multi INFO: found job 17001235 in provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:201 [INFO]  cancelling 17001235 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001235 on provider checkpoint
parsl.dataflow.dflow INFO: Shutting down executor multi
2024-03-11 23:03:05 proc_lsst.multi:40 [INFO]  Cancelling all provider resources
proc_lsst.multi INFO: Cancelling all provider resources
2024-03-11 23:03:05 proc_lsst.multi:47 [INFO]  new jobs since last cancel ['17001212', '17001213', '17001214', '17001215', '17001216', '17001217', '17001218', '17001219', '17001220', '17001221', '17001222', '17001223', '17001224', '17001225', '17001226', '17001227', '17001228', '17001229', '17001230', '17001231', '17001232', '17001233', '17001234', '17001235', '17001236', '17001237', '17001238', '17001240', '17001241', '17001242', '17001243', '17001244', '17001245', '17001246', '17001247', '17001248', '17001249', '17001157', '17001158', '17001159', '17001160', '17001161', '17001162', '17001163', '17001164', '17001165', '17001166', '17001167', '17001168', '17001169', '17001170', '17001171', '17001172', '17001173', '17001174', '17001175', '17001176', '17001177', '17001178', '17001179', '17001180', '17001181', '17001182', '17001183', '17001184', '17001185', '17001186', '17001192', '17001193', '17001194', '17001195', '17001196', '17001197', '17001198', '17001199', '17001200', '17001201', '17001202', '17001203', '17001204', '17001205', '17001206', '17001207', '17001208', '17001209', '17001210', '17001211', '17001187', '17001188', '17001189', '17001190', '17001191', '26021']
proc_lsst.multi INFO: new jobs since last cancel ['17001212', '17001213', '17001214', '17001215', '17001216', '17001217', '17001218', '17001219', '17001220', '17001221', '17001222', '17001223', '17001224', '17001225', '17001226', '17001227', '17001228', '17001229', '17001230', '17001231', '17001232', '17001233', '17001234', '17001235', '17001236', '17001237', '17001238', '17001240', '17001241', '17001242', '17001243', '17001244', '17001245', '17001246', '17001247', '17001248', '17001249', '17001157', '17001158', '17001159', '17001160', '17001161', '17001162', '17001163', '17001164', '17001165', '17001166', '17001167', '17001168', '17001169', '17001170', '17001171', '17001172', '17001173', '17001174', '17001175', '17001176', '17001177', '17001178', '17001179', '17001180', '17001181', '17001182', '17001183', '17001184', '17001185', '17001186', '17001192', '17001193', '17001194', '17001195', '17001196', '17001197', '17001198', '17001199', '17001200', '17001201', '17001202', '17001203', '17001204', '17001205', '17001206', '17001207', '17001208', '17001209', '17001210', '17001211', '17001187', '17001188', '17001189', '17001190', '17001191', '26021']
2024-03-11 23:03:05 proc_lsst.multi:146 [INFO]  found job 17001212 in provider checkpoint
proc_lsst.multi INFO: found job 17001212 in provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:201 [INFO]  cancelling 17001212 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001212 on provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:146 [INFO]  found job 17001213 in provider checkpoint
proc_lsst.multi INFO: found job 17001213 in provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:201 [INFO]  cancelling 17001213 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001213 on provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:146 [INFO]  found job 17001214 in provider checkpoint
proc_lsst.multi INFO: found job 17001214 in provider checkpoint
2024-03-11 23:03:05 proc_lsst.multi:201 [INFO]  cancelling 17001214 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001214 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001215 in provider checkpoint
proc_lsst.multi INFO: found job 17001215 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001215 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001215 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001216 in provider checkpoint
proc_lsst.multi INFO: found job 17001216 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001216 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001216 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001217 in provider checkpoint
proc_lsst.multi INFO: found job 17001217 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001217 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001217 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001218 in provider checkpoint
proc_lsst.multi INFO: found job 17001218 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001218 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001218 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001219 in provider checkpoint
proc_lsst.multi INFO: found job 17001219 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001219 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001219 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001220 in provider checkpoint
proc_lsst.multi INFO: found job 17001220 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001220 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001220 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001221 in provider checkpoint
proc_lsst.multi INFO: found job 17001221 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001221 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001221 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001222 in provider checkpoint
proc_lsst.multi INFO: found job 17001222 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001222 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001222 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001223 in provider checkpoint
proc_lsst.multi INFO: found job 17001223 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001223 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001223 on provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:146 [INFO]  found job 17001224 in provider checkpoint
proc_lsst.multi INFO: found job 17001224 in provider checkpoint
2024-03-11 23:03:06 proc_lsst.multi:201 [INFO]  cancelling 17001224 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001224 on provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:146 [INFO]  found job 17001225 in provider checkpoint
proc_lsst.multi INFO: found job 17001225 in provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:201 [INFO]  cancelling 17001225 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001225 on provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:146 [INFO]  found job 17001226 in provider checkpoint
proc_lsst.multi INFO: found job 17001226 in provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:201 [INFO]  cancelling 17001226 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001226 on provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:146 [INFO]  found job 17001227 in provider checkpoint
proc_lsst.multi INFO: found job 17001227 in provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:201 [INFO]  cancelling 17001227 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001227 on provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:146 [INFO]  found job 17001228 in provider checkpoint
proc_lsst.multi INFO: found job 17001228 in provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:201 [INFO]  cancelling 17001228 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001228 on provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:146 [INFO]  found job 17001229 in provider checkpoint
proc_lsst.multi INFO: found job 17001229 in provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:201 [INFO]  cancelling 17001229 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001229 on provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:146 [INFO]  found job 17001230 in provider checkpoint
proc_lsst.multi INFO: found job 17001230 in provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:201 [INFO]  cancelling 17001230 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001230 on provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:146 [INFO]  found job 17001231 in provider checkpoint
proc_lsst.multi INFO: found job 17001231 in provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:201 [INFO]  cancelling 17001231 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001231 on provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:146 [INFO]  found job 17001232 in provider checkpoint
proc_lsst.multi INFO: found job 17001232 in provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:201 [INFO]  cancelling 17001232 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001232 on provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:146 [INFO]  found job 17001233 in provider checkpoint
proc_lsst.multi INFO: found job 17001233 in provider checkpoint
2024-03-11 23:03:07 proc_lsst.multi:201 [INFO]  cancelling 17001233 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001233 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001234 in provider checkpoint
proc_lsst.multi INFO: found job 17001234 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001234 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001234 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001235 in provider checkpoint
proc_lsst.multi INFO: found job 17001235 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001235 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001235 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001236 in provider checkpoint
proc_lsst.multi INFO: found job 17001236 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001236 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001236 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001237 in provider checkpoint
proc_lsst.multi INFO: found job 17001237 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001237 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001237 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001238 in provider checkpoint
proc_lsst.multi INFO: found job 17001238 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001238 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001238 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001240 in provider checkpoint
proc_lsst.multi INFO: found job 17001240 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001240 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001240 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001241 in provider checkpoint
proc_lsst.multi INFO: found job 17001241 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001241 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001241 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001242 in provider checkpoint
proc_lsst.multi INFO: found job 17001242 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001242 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001242 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001243 in provider checkpoint
proc_lsst.multi INFO: found job 17001243 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001243 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001243 on provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:146 [INFO]  found job 17001244 in provider checkpoint
proc_lsst.multi INFO: found job 17001244 in provider checkpoint
2024-03-11 23:03:08 proc_lsst.multi:201 [INFO]  cancelling 17001244 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001244 on provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:146 [INFO]  found job 17001245 in provider checkpoint
proc_lsst.multi INFO: found job 17001245 in provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:201 [INFO]  cancelling 17001245 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001245 on provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:146 [INFO]  found job 17001246 in provider checkpoint
proc_lsst.multi INFO: found job 17001246 in provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:201 [INFO]  cancelling 17001246 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001246 on provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:146 [INFO]  found job 17001247 in provider checkpoint
proc_lsst.multi INFO: found job 17001247 in provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:201 [INFO]  cancelling 17001247 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001247 on provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:146 [INFO]  found job 17001248 in provider checkpoint
proc_lsst.multi INFO: found job 17001248 in provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:201 [INFO]  cancelling 17001248 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001248 on provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:146 [INFO]  found job 17001249 in provider checkpoint
proc_lsst.multi INFO: found job 17001249 in provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:201 [INFO]  cancelling 17001249 on provider checkpoint
proc_lsst.multi INFO: cancelling 17001249 on provider checkpoint
2024-03-11 23:03:09 proc_lsst.multi:146 [INFO]  found job 17001157 in provider astro
proc_lsst.multi INFO: found job 17001157 in provider astro
2024-03-11 23:03:09 proc_lsst.multi:201 [INFO]  cancelling 17001157 on provider astro
proc_lsst.multi INFO: cancelling 17001157 on provider astro
2024-03-11 23:03:09 proc_lsst.multi:146 [INFO]  found job 17001158 in provider astro
proc_lsst.multi INFO: found job 17001158 in provider astro
2024-03-11 23:03:09 proc_lsst.multi:201 [INFO]  cancelling 17001158 on provider astro
proc_lsst.multi INFO: cancelling 17001158 on provider astro
2024-03-11 23:03:09 proc_lsst.multi:146 [INFO]  found job 17001159 in provider astro
proc_lsst.multi INFO: found job 17001159 in provider astro
2024-03-11 23:03:09 proc_lsst.multi:201 [INFO]  cancelling 17001159 on provider astro
proc_lsst.multi INFO: cancelling 17001159 on provider astro
2024-03-11 23:03:09 proc_lsst.multi:146 [INFO]  found job 17001160 in provider astro
proc_lsst.multi INFO: found job 17001160 in provider astro
2024-03-11 23:03:09 proc_lsst.multi:201 [INFO]  cancelling 17001160 on provider astro
proc_lsst.multi INFO: cancelling 17001160 on provider astro
2024-03-11 23:03:10 proc_lsst.multi:146 [INFO]  found job 17001161 in provider astro
proc_lsst.multi INFO: found job 17001161 in provider astro
2024-03-11 23:03:10 proc_lsst.multi:201 [INFO]  cancelling 17001161 on provider astro
proc_lsst.multi INFO: cancelling 17001161 on provider astro
2024-03-11 23:03:10 proc_lsst.multi:146 [INFO]  found job 17001162 in provider astro
proc_lsst.multi INFO: found job 17001162 in provider astro
2024-03-11 23:03:10 proc_lsst.multi:201 [INFO]  cancelling 17001162 on provider astro
proc_lsst.multi INFO: cancelling 17001162 on provider astro
2024-03-11 23:03:10 proc_lsst.multi:146 [INFO]  found job 17001163 in provider astro
proc_lsst.multi INFO: found job 17001163 in provider astro
2024-03-11 23:03:10 proc_lsst.multi:201 [INFO]  cancelling 17001163 on provider astro
proc_lsst.multi INFO: cancelling 17001163 on provider astro
2024-03-11 23:03:10 proc_lsst.multi:146 [INFO]  found job 17001164 in provider astro
proc_lsst.multi INFO: found job 17001164 in provider astro
2024-03-11 23:03:10 proc_lsst.multi:201 [INFO]  cancelling 17001164 on provider astro
proc_lsst.multi INFO: cancelling 17001164 on provider astro
2024-03-11 23:03:10 proc_lsst.multi:146 [INFO]  found job 17001165 in provider astro
proc_lsst.multi INFO: found job 17001165 in provider astro
2024-03-11 23:03:10 proc_lsst.multi:201 [INFO]  cancelling 17001165 on provider astro
proc_lsst.multi INFO: cancelling 17001165 on provider astro
2024-03-11 23:03:10 proc_lsst.multi:146 [INFO]  found job 17001166 in provider astro
proc_lsst.multi INFO: found job 17001166 in provider astro
2024-03-11 23:03:10 proc_lsst.multi:201 [INFO]  cancelling 17001166 on provider astro
proc_lsst.multi INFO: cancelling 17001166 on provider astro
2024-03-11 23:03:10 proc_lsst.multi:146 [INFO]  found job 17001167 in provider astro
proc_lsst.multi INFO: found job 17001167 in provider astro
2024-03-11 23:03:10 proc_lsst.multi:201 [INFO]  cancelling 17001167 on provider astro
proc_lsst.multi INFO: cancelling 17001167 on provider astro
2024-03-11 23:03:10 proc_lsst.multi:146 [INFO]  found job 17001168 in provider astro
proc_lsst.multi INFO: found job 17001168 in provider astro
2024-03-11 23:03:10 proc_lsst.multi:201 [INFO]  cancelling 17001168 on provider astro
proc_lsst.multi INFO: cancelling 17001168 on provider astro
2024-03-11 23:03:10 proc_lsst.multi:146 [INFO]  found job 17001169 in provider astro
proc_lsst.multi INFO: found job 17001169 in provider astro
2024-03-11 23:03:10 proc_lsst.multi:201 [INFO]  cancelling 17001169 on provider astro
proc_lsst.multi INFO: cancelling 17001169 on provider astro
2024-03-11 23:03:11 proc_lsst.multi:146 [INFO]  found job 17001170 in provider astro
proc_lsst.multi INFO: found job 17001170 in provider astro
2024-03-11 23:03:11 proc_lsst.multi:201 [INFO]  cancelling 17001170 on provider astro
proc_lsst.multi INFO: cancelling 17001170 on provider astro
2024-03-11 23:03:11 proc_lsst.multi:146 [INFO]  found job 17001171 in provider astro
proc_lsst.multi INFO: found job 17001171 in provider astro
2024-03-11 23:03:11 proc_lsst.multi:201 [INFO]  cancelling 17001171 on provider astro
proc_lsst.multi INFO: cancelling 17001171 on provider astro
2024-03-11 23:03:11 proc_lsst.multi:146 [INFO]  found job 17001172 in provider astro
proc_lsst.multi INFO: found job 17001172 in provider astro
2024-03-11 23:03:11 proc_lsst.multi:201 [INFO]  cancelling 17001172 on provider astro
proc_lsst.multi INFO: cancelling 17001172 on provider astro
2024-03-11 23:03:11 proc_lsst.multi:146 [INFO]  found job 17001173 in provider astro
proc_lsst.multi INFO: found job 17001173 in provider astro
2024-03-11 23:03:11 proc_lsst.multi:201 [INFO]  cancelling 17001173 on provider astro
proc_lsst.multi INFO: cancelling 17001173 on provider astro
2024-03-11 23:03:11 proc_lsst.multi:146 [INFO]  found job 17001174 in provider astro
proc_lsst.multi INFO: found job 17001174 in provider astro
2024-03-11 23:03:11 proc_lsst.multi:201 [INFO]  cancelling 17001174 on provider astro
proc_lsst.multi INFO: cancelling 17001174 on provider astro
2024-03-11 23:03:11 proc_lsst.multi:146 [INFO]  found job 17001175 in provider astro
proc_lsst.multi INFO: found job 17001175 in provider astro
2024-03-11 23:03:11 proc_lsst.multi:201 [INFO]  cancelling 17001175 on provider astro
proc_lsst.multi INFO: cancelling 17001175 on provider astro
2024-03-11 23:03:11 proc_lsst.multi:146 [INFO]  found job 17001176 in provider astro
proc_lsst.multi INFO: found job 17001176 in provider astro
2024-03-11 23:03:11 proc_lsst.multi:201 [INFO]  cancelling 17001176 on provider astro
proc_lsst.multi INFO: cancelling 17001176 on provider astro
2024-03-11 23:03:11 proc_lsst.multi:146 [INFO]  found job 17001177 in provider astro
proc_lsst.multi INFO: found job 17001177 in provider astro
2024-03-11 23:03:11 proc_lsst.multi:201 [INFO]  cancelling 17001177 on provider astro
proc_lsst.multi INFO: cancelling 17001177 on provider astro
2024-03-11 23:03:12 proc_lsst.multi:146 [INFO]  found job 17001178 in provider astro
proc_lsst.multi INFO: found job 17001178 in provider astro
2024-03-11 23:03:12 proc_lsst.multi:201 [INFO]  cancelling 17001178 on provider astro
proc_lsst.multi INFO: cancelling 17001178 on provider astro
2024-03-11 23:03:12 proc_lsst.multi:146 [INFO]  found job 17001179 in provider astro
proc_lsst.multi INFO: found job 17001179 in provider astro
2024-03-11 23:03:12 proc_lsst.multi:201 [INFO]  cancelling 17001179 on provider astro
proc_lsst.multi INFO: cancelling 17001179 on provider astro
2024-03-11 23:03:12 proc_lsst.multi:146 [INFO]  found job 17001180 in provider astro
proc_lsst.multi INFO: found job 17001180 in provider astro
2024-03-11 23:03:12 proc_lsst.multi:201 [INFO]  cancelling 17001180 on provider astro
proc_lsst.multi INFO: cancelling 17001180 on provider astro
2024-03-11 23:03:12 proc_lsst.multi:146 [INFO]  found job 17001181 in provider astro
proc_lsst.multi INFO: found job 17001181 in provider astro
2024-03-11 23:03:12 proc_lsst.multi:201 [INFO]  cancelling 17001181 on provider astro
proc_lsst.multi INFO: cancelling 17001181 on provider astro
2024-03-11 23:03:12 proc_lsst.multi:146 [INFO]  found job 17001182 in provider astro
proc_lsst.multi INFO: found job 17001182 in provider astro
2024-03-11 23:03:12 proc_lsst.multi:201 [INFO]  cancelling 17001182 on provider astro
proc_lsst.multi INFO: cancelling 17001182 on provider astro
2024-03-11 23:03:12 proc_lsst.multi:146 [INFO]  found job 17001183 in provider astro
proc_lsst.multi INFO: found job 17001183 in provider astro
2024-03-11 23:03:12 proc_lsst.multi:201 [INFO]  cancelling 17001183 on provider astro
proc_lsst.multi INFO: cancelling 17001183 on provider astro
2024-03-11 23:03:12 proc_lsst.multi:146 [INFO]  found job 17001184 in provider astro
proc_lsst.multi INFO: found job 17001184 in provider astro
2024-03-11 23:03:12 proc_lsst.multi:201 [INFO]  cancelling 17001184 on provider astro
proc_lsst.multi INFO: cancelling 17001184 on provider astro
2024-03-11 23:03:12 proc_lsst.multi:146 [INFO]  found job 17001185 in provider astro
proc_lsst.multi INFO: found job 17001185 in provider astro
2024-03-11 23:03:12 proc_lsst.multi:201 [INFO]  cancelling 17001185 on provider astro
proc_lsst.multi INFO: cancelling 17001185 on provider astro
2024-03-11 23:03:12 proc_lsst.multi:146 [INFO]  found job 17001186 in provider astro
proc_lsst.multi INFO: found job 17001186 in provider astro
2024-03-11 23:03:12 proc_lsst.multi:201 [INFO]  cancelling 17001186 on provider astro
proc_lsst.multi INFO: cancelling 17001186 on provider astro
2024-03-11 23:03:13 proc_lsst.multi:146 [INFO]  found job 17001192 in provider escience_a40
proc_lsst.multi INFO: found job 17001192 in provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:201 [INFO]  cancelling 17001192 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001192 on provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:146 [INFO]  found job 17001193 in provider escience_a40
proc_lsst.multi INFO: found job 17001193 in provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:201 [INFO]  cancelling 17001193 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001193 on provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:146 [INFO]  found job 17001194 in provider escience_a40
proc_lsst.multi INFO: found job 17001194 in provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:201 [INFO]  cancelling 17001194 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001194 on provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:146 [INFO]  found job 17001195 in provider escience_a40
proc_lsst.multi INFO: found job 17001195 in provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:201 [INFO]  cancelling 17001195 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001195 on provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:146 [INFO]  found job 17001196 in provider escience_a40
proc_lsst.multi INFO: found job 17001196 in provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:201 [INFO]  cancelling 17001196 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001196 on provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:146 [INFO]  found job 17001197 in provider escience_a40
proc_lsst.multi INFO: found job 17001197 in provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:201 [INFO]  cancelling 17001197 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001197 on provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:146 [INFO]  found job 17001198 in provider escience_a40
proc_lsst.multi INFO: found job 17001198 in provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:201 [INFO]  cancelling 17001198 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001198 on provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:146 [INFO]  found job 17001199 in provider escience_a40
proc_lsst.multi INFO: found job 17001199 in provider escience_a40
2024-03-11 23:03:13 proc_lsst.multi:201 [INFO]  cancelling 17001199 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001199 on provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:146 [INFO]  found job 17001200 in provider escience_a40
proc_lsst.multi INFO: found job 17001200 in provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:201 [INFO]  cancelling 17001200 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001200 on provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:146 [INFO]  found job 17001201 in provider escience_a40
proc_lsst.multi INFO: found job 17001201 in provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:201 [INFO]  cancelling 17001201 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001201 on provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:146 [INFO]  found job 17001202 in provider escience_a40
proc_lsst.multi INFO: found job 17001202 in provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:201 [INFO]  cancelling 17001202 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001202 on provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:146 [INFO]  found job 17001203 in provider escience_a40
proc_lsst.multi INFO: found job 17001203 in provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:201 [INFO]  cancelling 17001203 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001203 on provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:146 [INFO]  found job 17001204 in provider escience_a40
proc_lsst.multi INFO: found job 17001204 in provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:201 [INFO]  cancelling 17001204 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001204 on provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:146 [INFO]  found job 17001205 in provider escience_a40
proc_lsst.multi INFO: found job 17001205 in provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:201 [INFO]  cancelling 17001205 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001205 on provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:146 [INFO]  found job 17001206 in provider escience_a40
proc_lsst.multi INFO: found job 17001206 in provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:201 [INFO]  cancelling 17001206 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001206 on provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:146 [INFO]  found job 17001207 in provider escience_a40
proc_lsst.multi INFO: found job 17001207 in provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:201 [INFO]  cancelling 17001207 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001207 on provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:146 [INFO]  found job 17001208 in provider escience_a40
proc_lsst.multi INFO: found job 17001208 in provider escience_a40
2024-03-11 23:03:14 proc_lsst.multi:201 [INFO]  cancelling 17001208 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001208 on provider escience_a40
2024-03-11 23:03:15 proc_lsst.multi:146 [INFO]  found job 17001209 in provider escience_a40
proc_lsst.multi INFO: found job 17001209 in provider escience_a40
2024-03-11 23:03:15 proc_lsst.multi:201 [INFO]  cancelling 17001209 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001209 on provider escience_a40
2024-03-11 23:03:15 proc_lsst.multi:146 [INFO]  found job 17001210 in provider escience_a40
proc_lsst.multi INFO: found job 17001210 in provider escience_a40
2024-03-11 23:03:15 proc_lsst.multi:201 [INFO]  cancelling 17001210 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001210 on provider escience_a40
2024-03-11 23:03:15 proc_lsst.multi:146 [INFO]  found job 17001211 in provider escience_a40
proc_lsst.multi INFO: found job 17001211 in provider escience_a40
2024-03-11 23:03:15 proc_lsst.multi:201 [INFO]  cancelling 17001211 on provider escience_a40
proc_lsst.multi INFO: cancelling 17001211 on provider escience_a40
2024-03-11 23:03:15 proc_lsst.multi:146 [INFO]  found job 17001187 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001187 in provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:201 [INFO]  cancelling 17001187 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17001187 on provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:146 [INFO]  found job 17001188 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001188 in provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:201 [INFO]  cancelling 17001188 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17001188 on provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:146 [INFO]  found job 17001189 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001189 in provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:201 [INFO]  cancelling 17001189 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17001189 on provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:146 [INFO]  found job 17001190 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001190 in provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:201 [INFO]  cancelling 17001190 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17001190 on provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:146 [INFO]  found job 17001191 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17001191 in provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:201 [INFO]  cancelling 17001191 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17001191 on provider escience_rtx6k
2024-03-11 23:03:15 proc_lsst.multi:146 [INFO]  found job 26021 in provider local
proc_lsst.multi INFO: found job 26021 in provider local
2024-03-11 23:03:15 proc_lsst.multi:201 [INFO]  cancelling 26021 on provider local
proc_lsst.multi INFO: cancelling 26021 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 26021
parsl.providers.local.local WARNING: Failed to kill PID: 26021 and child processes on local
2024-03-11 23:03:17 proc_lsst.multi:50 [INFO]  no new jobs since last cancel, resuming executor shutdown
proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown
parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown
parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt
parsl.dataflow.dflow INFO: Shut down executor multi
parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal
parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete
parsl.executors.threads DEBUG: Done with executor shutdown
parsl.dataflow.dflow INFO: Shut down executor _parsl_internal
parsl.dataflow.dflow INFO: Terminated executors
parsl.dataflow.dflow INFO: DFK cleanup complete
parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread
lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 197.0063 seconds
lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190926_B1b_science#step0_20240311T225937Z' submitted for execution with id 'None'
lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 197.0180 seconds; current memory usage: 0.281 Gibyte, delta: 0.004 Gibyte, peak delta: 0.007 Gibyte
lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 220.1008 seconds; current memory usage: 0.281 Gibyte, delta: 0.142 Gibyte, peak delta: 0.146 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.284 Gibyte (main), 0.284 Gibyte (largest child process)
Run Id: None
Run Name: DEEP_20190926_B1b_science#step0_20240311T225937Z
parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up

real	3m42.288s
user	0m54.014s
sys	0m28.193s