+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit
+ bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190707/B0b --output-run DEEP/20190707/B0b/science#step0/20240310T192404Z --qgraph pipeline.qgraph
lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process.
lsst.ctrl.bps.drivers INFO: Starting submission process
lsst.ctrl.bps.drivers INFO: Initializing execution environment
lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 6.3175 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph)
lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0045 seconds
lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0073 seconds
lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/pipeline_orig.qgraph DEEP/20190707/B0b/science#step0/20240310T192404Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/pipeline.qgraph 
lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 3.6995 seconds
lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 9.6886 seconds; current memory usage: 0.270 Gibyte, delta: 0.092 Gibyte, peak delta: 0.080 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.263 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs)
lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0164 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.263 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 186 cluster(s)
lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow)
lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190707_B0b_science#step0_20240310T192404Z'
lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0740 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.263 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 188 job(s) (including final)
lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow)
parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses
    s_addresses.add(address_by_interface(interface))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface
    return socket.inet_ntoa(fcntl.ioctl(
                            ^^^^^^^^^^^^
OSError: [Errno 99] Cannot assign requested address
lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z
lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.7379 seconds; current memory usage: 0.271 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.271 Gibyte (main), 0.263 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting submit stage
lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution
parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000
parsl.dataflow.dflow INFO: Starting DataFlowKernel with config
Config(
    app_cache=True, 
    checkpoint_files=None, 
    checkpoint_mode='task_exit', 
    checkpoint_period=None, 
    executors=[MultiHighThroughputExecutor()], 
    garbage_collect=True, 
    initialize_logging=True, 
    internal_tasks_max_threads=10, 
    max_idletime=120.0, 
    monitoring=None, 
    retries=1, 
    retry_handler=None, 
    run_dir='runinfo', 
    strategy='simple', 
    usage_tracking=False
)
parsl.dataflow.dflow INFO: Parsl version: 2023.06.12
parsl.usage_tracking.usage DEBUG: Tracking status: False
parsl.dataflow.dflow INFO: Run id is: 4ff12e90-1675-4655-9f16-39fa289ecf05
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py
parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name
parsl.dataflow.memoization INFO: App caching initialized
parsl.dataflow.strategy DEBUG: Scaling strategy: simple
parsl.executors.high_throughput.executor DEBUG: Starting queue management thread
parsl.executors.high_throughput.executor DEBUG: queue management worker starting
parsl.executors.high_throughput.executor DEBUG: Started queue management thread
Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z
parsl.executors.high_throughput.executor DEBUG: Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 23352441894656)>
parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider:
<proc_lsst.multi.MultiProvider object at 0x153d2b507590>
parsl.executors.status_handling INFO: Scaling out by 1 blocks
parsl.executors.status_handling INFO: Allocated block ID 0
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0
2024-03-10 19:24:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
2024-03-10 19:24:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-10 19:24:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
parsl.providers.local.local DEBUG: Launching in remote mode
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
2024-03-10 19:24:43 proc_lsst.multi:170 [INFO]  [multi] job_id 96597
proc_lsst.multi INFO: [multi] job_id 96597
2024-03-10 19:24:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:43 proc_lsst.multi:178 [INFO]  [multi] provider local accepted submit and returned 96597
proc_lsst.multi INFO: [multi] provider local accepted submit and returned 96597
parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 96597
parsl.dataflow.job_status_poller DEBUG: Adding executor multi
parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: <AppFuture at 0x153d3597bb10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 622a7f851754d3d31dadc87b23caa535
parsl.dataflow.memoization INFO: Task 0 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/13d2c775-ec0e-40ea-be1c-f93f71d524ab_isrForCrosstalkSources_871265_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/13d2c775-ec0e-40ea-be1c-f93f71d524ab_isrForCrosstalkSources_871265_49.stderr
parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: <AppFuture at 0x153d2b4f4750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 20802451ad18f55d11ca03a36a25bc0e
parsl.dataflow.memoization INFO: Task 1 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e3ad8990-1aae-4162-9f21-f2fd0cde30ba_isrForCrosstalkSources_871267_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e3ad8990-1aae-4162-9f21-f2fd0cde30ba_isrForCrosstalkSources_871267_33.stderr
parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: <AppFuture at 0x153d359b3e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 5666406ad03074ad8d6b2a1cac783fd4
parsl.dataflow.memoization INFO: Task 2 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2679c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0a48706a-caa3-47e7-82cd-be996e44fcc9_isrForCrosstalkSources_871267_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0a48706a-caa3-47e7-82cd-be996e44fcc9_isrForCrosstalkSources_871267_47.stderr
parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: <AppFuture at 0x153d2b49f490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 5e9c17f39e37a1a7f9a0d6ace0c7dcbc
parsl.dataflow.memoization INFO: Task 3 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fbbcbd9e-646c-4954-8a11-72efa5d4cd2d_isrForCrosstalkSources_871267_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fbbcbd9e-646c-4954-8a11-72efa5d4cd2d_isrForCrosstalkSources_871267_4.stderr
parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: <AppFuture at 0x153d2b487010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 5cbf7d4b5b1bdf075d42e12ec8d5640c
parsl.dataflow.memoization INFO: Task 4 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/03d07d67-be23-4de3-a5c9-8c2d492f38f6_isrForCrosstalkSources_871265_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/03d07d67-be23-4de3-a5c9-8c2d492f38f6_isrForCrosstalkSources_871265_59.stderr
parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: <AppFuture at 0x153d2b486150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 5 has memoization hash e209a5feb60b804a9b716d61e340b379
parsl.dataflow.memoization INFO: Task 5 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/47221143-6556-4824-8759-69baad029ede_isrForCrosstalkSources_871266_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/47221143-6556-4824-8759-69baad029ede_isrForCrosstalkSources_871266_44.stderr
parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: <AppFuture at 0x153d2b4721d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 6 has memoization hash dba3f03c5a9c6f7a526617187f9d40f1
parsl.dataflow.memoization INFO: Task 6 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b8c90a3c-2439-475a-89ad-d0621b8ae117_isrForCrosstalkSources_871267_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b8c90a3c-2439-475a-89ad-d0621b8ae117_isrForCrosstalkSources_871267_39.stderr
parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: <AppFuture at 0x153d2b470b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 87e3a02195b6721180d5177611e24407
parsl.dataflow.memoization INFO: Task 7 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b85b4baa-d9b4-47b0-a6cc-0d0e9ca12a68_isrForCrosstalkSources_871267_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b85b4baa-d9b4-47b0-a6cc-0d0e9ca12a68_isrForCrosstalkSources_871267_57.stderr
parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: <AppFuture at 0x153d2b471d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 5360b5562c9970d4ec9103a6ffd66b51
parsl.dataflow.memoization INFO: Task 8 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b266de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4c7491e8-dfa4-457c-bcfa-c16a9df0ca26_isrForCrosstalkSources_871266_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4c7491e8-dfa4-457c-bcfa-c16a9df0ca26_isrForCrosstalkSources_871266_42.stderr
parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: <AppFuture at 0x153d2b622690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 14ecff33bb36d01bf6d1f129b7ec0c41
parsl.dataflow.memoization INFO: Task 9 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/c3fb05c4-3faa-4932-88f6-63b9b997865f_isrForCrosstalkSources_871266_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/c3fb05c4-3faa-4932-88f6-63b9b997865f_isrForCrosstalkSources_871266_1.stderr
parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: <AppFuture at 0x153d2b468d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 10 has memoization hash be058a1ebce44bb312170f2017a57231
parsl.dataflow.memoization INFO: Task 10 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e83c2e36-fb6c-42fe-a0cc-3d8ea05217f6_isrForCrosstalkSources_871266_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e83c2e36-fb6c-42fe-a0cc-3d8ea05217f6_isrForCrosstalkSources_871266_62.stderr
parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: <AppFuture at 0x153d2b468150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 81877b80a91a5249f966dfddbc2ddf64
parsl.dataflow.memoization INFO: Task 11 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b267e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/13db6717-de90-4eac-991c-e804725090bb_isrForCrosstalkSources_871267_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/13db6717-de90-4eac-991c-e804725090bb_isrForCrosstalkSources_871267_59.stderr
parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: <AppFuture at 0x153d2b4685d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 349e2e2963cdff8af49096b96200d178
parsl.dataflow.memoization INFO: Task 12 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/faba3ec8-cf43-4b0d-a560-74009baedb7e_isrForCrosstalkSources_871267_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/faba3ec8-cf43-4b0d-a560-74009baedb7e_isrForCrosstalkSources_871267_23.stderr
parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: <AppFuture at 0x153d2b489790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 13 has memoization hash eeb64e015b80989f66df0837b1a30162
parsl.dataflow.memoization INFO: Task 13 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b73a0d2b-c48e-4b39-852a-f774a61fbc32_isrForCrosstalkSources_871267_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b73a0d2b-c48e-4b39-852a-f774a61fbc32_isrForCrosstalkSources_871267_55.stderr
parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: <AppFuture at 0x153d2b45dc10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 86ca76c247780df5a14dd5c5a6b82274
parsl.dataflow.memoization INFO: Task 14 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/44ab5d5c-8386-4229-b685-1c8467dcacce_isrForCrosstalkSources_871267_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/44ab5d5c-8386-4229-b685-1c8467dcacce_isrForCrosstalkSources_871267_28.stderr
parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: <AppFuture at 0x153d2b45fc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 0cbd4b89b5d4fd575578420b1e9fbfeb
parsl.dataflow.memoization INFO: Task 15 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5eefb463-97ba-4476-86de-3329a2dffc63_isrForCrosstalkSources_871266_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5eefb463-97ba-4476-86de-3329a2dffc63_isrForCrosstalkSources_871266_58.stderr
parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: <AppFuture at 0x153d2b45f4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 16 has memoization hash e4f153eac18e87c36d60e5af6417e4b9
parsl.dataflow.memoization INFO: Task 16 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b644ffa9-c400-499c-a86d-7319e6eaab21_isrForCrosstalkSources_871266_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b644ffa9-c400-499c-a86d-7319e6eaab21_isrForCrosstalkSources_871266_16.stderr
parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: <AppFuture at 0x153d2b454250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 47a17750b1bd769c146b992d9d811235
parsl.dataflow.memoization INFO: Task 17 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b2b7cabe-0d5e-4420-9b32-74e0d4b99f02_isrForCrosstalkSources_871265_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b2b7cabe-0d5e-4420-9b32-74e0d4b99f02_isrForCrosstalkSources_871265_23.stderr
parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: <AppFuture at 0x153d2b455f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 6ce277dae347b0a32c614c7bf94efd38
parsl.dataflow.memoization INFO: Task 18 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/659599ca-b49c-486c-be8f-6b2b1c8de4b4_isrForCrosstalkSources_871265_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/659599ca-b49c-486c-be8f-6b2b1c8de4b4_isrForCrosstalkSources_871265_45.stderr
parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: <AppFuture at 0x153d2b4fc610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 19 has memoization hash e7bdca0cf95555fdcfd035228b098278
parsl.dataflow.memoization INFO: Task 19 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/7d41a93a-b0b5-48b0-92cc-2f98c606b294_isrForCrosstalkSources_871266_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/7d41a93a-b0b5-48b0-92cc-2f98c606b294_isrForCrosstalkSources_871266_57.stderr
parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: <AppFuture at 0x153d2b44a090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 7fabd2050023c3e0346511df492965aa
parsl.dataflow.memoization INFO: Task 20 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d8b3def0-8bcf-42f2-a155-4bf92860bc6e_isrForCrosstalkSources_871267_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d8b3def0-8bcf-42f2-a155-4bf92860bc6e_isrForCrosstalkSources_871267_26.stderr
parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: <AppFuture at 0x153d2b44b250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 5914bd0f7b8f91bc9ad485510977eb05
parsl.dataflow.memoization INFO: Task 21 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/cef70236-a502-4e38-bf44-b2a326743040_isrForCrosstalkSources_871267_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/cef70236-a502-4e38-bf44-b2a326743040_isrForCrosstalkSources_871267_27.stderr
parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: <AppFuture at 0x153d2b44a610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 1fd4aaeb2f945cf30ba3f4fc0e1b568e
parsl.dataflow.memoization INFO: Task 22 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4ceb96ca-36c0-490d-ae8a-5c875f3809be_isrForCrosstalkSources_871266_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4ceb96ca-36c0-490d-ae8a-5c875f3809be_isrForCrosstalkSources_871266_25.stderr
parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: <AppFuture at 0x153d2b45f910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 23 has memoization hash dd7ac59c6d512468c7b0881ac20498df
parsl.dataflow.memoization INFO: Task 23 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a4d77a1e-83a6-4543-9058-27bdb014abfa_isrForCrosstalkSources_871266_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a4d77a1e-83a6-4543-9058-27bdb014abfa_isrForCrosstalkSources_871266_29.stderr
parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: <AppFuture at 0x153d2b444e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 24 has memoization hash cae0c782add39d3da6ed7047d8be3445
parsl.dataflow.memoization INFO: Task 24 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a9852d3a-299f-4d1c-a38b-e64023853732_isrForCrosstalkSources_871266_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a9852d3a-299f-4d1c-a38b-e64023853732_isrForCrosstalkSources_871266_54.stderr
parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: <AppFuture at 0x153d2b445690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 25 has memoization hash c665ced78181a0161521fb006f8d99a3
parsl.dataflow.memoization INFO: Task 25 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/3fc7e56b-072d-4635-9139-5d61a19b6e86_isrForCrosstalkSources_871267_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/3fc7e56b-072d-4635-9139-5d61a19b6e86_isrForCrosstalkSources_871267_11.stderr
parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: <AppFuture at 0x153d2b457250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 26 has memoization hash ab9a8454445ff502a6b1cb39dfef223f
parsl.dataflow.memoization INFO: Task 26 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/97378aeb-8266-46dc-bea0-4cb29c5c8d38_isrForCrosstalkSources_871267_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/97378aeb-8266-46dc-bea0-4cb29c5c8d38_isrForCrosstalkSources_871267_24.stderr
parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: <AppFuture at 0x153d2b43e710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 27 has memoization hash c4c1b0bd2c48d1e86c20fcdc805af577
parsl.dataflow.memoization INFO: Task 27 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c78ce7ec-9f64-423a-b920-d8b0cc456a42_isrForCrosstalkSources_871265_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c78ce7ec-9f64-423a-b920-d8b0cc456a42_isrForCrosstalkSources_871265_4.stderr
parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: <AppFuture at 0x153d2b43f050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 0ff7d2f7116d9308c0e35deb2f7b3061
parsl.dataflow.memoization INFO: Task 28 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/e85a2a5c-09ca-4ae8-987d-60b0d5cd14d0_isrForCrosstalkSources_871265_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/e85a2a5c-09ca-4ae8-987d-60b0d5cd14d0_isrForCrosstalkSources_871265_5.stderr
parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: <AppFuture at 0x153d2b43dd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 71cc386eb3d030f683d5a853e21334ec
parsl.dataflow.memoization INFO: Task 29 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3c63998c-c5c7-4de1-aad3-049914f4bd32_isrForCrosstalkSources_871265_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3c63998c-c5c7-4de1-aad3-049914f4bd32_isrForCrosstalkSources_871265_15.stderr
parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: <AppFuture at 0x153d2b43d250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 30 has memoization hash d0f305278d45dae49ff3fba9bb3532bc
parsl.dataflow.memoization INFO: Task 30 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e9149a42-e609-4bb6-86b8-fb2064946e62_isrForCrosstalkSources_871266_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e9149a42-e609-4bb6-86b8-fb2064946e62_isrForCrosstalkSources_871266_10.stderr
parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: <AppFuture at 0x153d2b637f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 02eb55797463267df4f5191d40cc4d5b
parsl.dataflow.memoization INFO: Task 31 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/9cdb5abc-10e9-4d90-9b2e-402c82c5d5a8_isrForCrosstalkSources_871265_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/9cdb5abc-10e9-4d90-9b2e-402c82c5d5a8_isrForCrosstalkSources_871265_18.stderr
parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: <AppFuture at 0x153d2b634e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 32a579fe8afb95e0b652a1da19ce0383
parsl.dataflow.memoization INFO: Task 32 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/44a219bc-0e48-4af1-93d3-5c75d1beccbb_isrForCrosstalkSources_871265_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/44a219bc-0e48-4af1-93d3-5c75d1beccbb_isrForCrosstalkSources_871265_46.stderr
parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: <AppFuture at 0x153d2b637ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 33 has memoization hash e5d3b1e5f16f13a87d09bce653727ef2
parsl.dataflow.memoization INFO: Task 33 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4d835c42-7f83-4ff0-8f73-0ff0c8251ec4_isrForCrosstalkSources_871265_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4d835c42-7f83-4ff0-8f73-0ff0c8251ec4_isrForCrosstalkSources_871265_16.stderr
parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: <AppFuture at 0x153d2b634cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 34 has memoization hash a3a2d9d221ec5954320da89da95e5500
parsl.dataflow.memoization INFO: Task 34 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/daaad190-3862-48f3-88ce-aa1025df228b_isrForCrosstalkSources_871267_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/daaad190-3862-48f3-88ce-aa1025df228b_isrForCrosstalkSources_871267_17.stderr
parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: <AppFuture at 0x153d2b61e710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 976bba566082ef6e85ebd8ea59e18f99
parsl.dataflow.memoization INFO: Task 35 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/6fc2c924-b88d-4fc9-8027-8462c8bf8bf8_isrForCrosstalkSources_871267_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/6fc2c924-b88d-4fc9-8027-8462c8bf8bf8_isrForCrosstalkSources_871267_62.stderr
parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: <AppFuture at 0x153d2b61d1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 6e04a6932acdb9e158965ee8f2da6f8d
parsl.dataflow.memoization INFO: Task 36 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/de632ea0-cee2-403b-b865-61bae1123a50_isrForCrosstalkSources_871266_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/de632ea0-cee2-403b-b865-61bae1123a50_isrForCrosstalkSources_871266_36.stderr
parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: <AppFuture at 0x153d2b628ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 0cdc79a99d4618cf619017fab8621bc9
parsl.dataflow.memoization INFO: Task 37 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/aa3f36bd-2c84-43a1-a3b5-ed4e3e2e2336_isrForCrosstalkSources_871265_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/aa3f36bd-2c84-43a1-a3b5-ed4e3e2e2336_isrForCrosstalkSources_871265_55.stderr
parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: <AppFuture at 0x153d2b62a250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 1fc113d3bafe0ebec05c124432b5c81f
parsl.dataflow.memoization INFO: Task 38 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/974305a6-7d74-49b6-b8ef-9711123ce494_isrForCrosstalkSources_871265_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/974305a6-7d74-49b6-b8ef-9711123ce494_isrForCrosstalkSources_871265_35.stderr
parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: <AppFuture at 0x153d2b6d5a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 31077c751aad9300a18a8e0260d485b8
parsl.dataflow.memoization INFO: Task 39 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/54ae2448-a97d-4df2-9ae7-483c2b7d90c6_isrForCrosstalkSources_871266_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/54ae2448-a97d-4df2-9ae7-483c2b7d90c6_isrForCrosstalkSources_871266_23.stderr
parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: <AppFuture at 0x153d2b615d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 40 has memoization hash eec5814afbf7635568eb11825d5749f4
parsl.dataflow.memoization INFO: Task 40 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/42a075bd-db97-42ce-ad23-5bddb897bd72_isrForCrosstalkSources_871265_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/42a075bd-db97-42ce-ad23-5bddb897bd72_isrForCrosstalkSources_871265_42.stderr
parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: <AppFuture at 0x153d2b637bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 327523c011be0e97bf4ad69389b428f7
parsl.dataflow.memoization INFO: Task 41 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8dac5e52-ad75-44a9-9548-b6f0a75ca510_isrForCrosstalkSources_871265_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8dac5e52-ad75-44a9-9548-b6f0a75ca510_isrForCrosstalkSources_871265_22.stderr
parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: <AppFuture at 0x153d2b23fcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 84bcabaa05891415fa447d2c617dcf43
parsl.dataflow.memoization INFO: Task 42 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fa5f097f-67bb-4061-8c53-981c0dc00337_isrForCrosstalkSources_871267_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fa5f097f-67bb-4061-8c53-981c0dc00337_isrForCrosstalkSources_871267_54.stderr
parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: <AppFuture at 0x153d2b23dc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 43 has memoization hash c0affca01f280a332abbcc8c6d4ac7c0
parsl.dataflow.memoization INFO: Task 43 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/33a35797-b2fe-4dc8-8196-177a94d65e3e_isrForCrosstalkSources_871267_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/33a35797-b2fe-4dc8-8196-177a94d65e3e_isrForCrosstalkSources_871267_56.stderr
parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: <AppFuture at 0x153d2b23fd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 96d45e0829ccdbcfa3824f105658257a
parsl.dataflow.memoization INFO: Task 44 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/af400b6e-c262-4895-a4bc-a59eb01ab5b8_isrForCrosstalkSources_871267_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/af400b6e-c262-4895-a4bc-a59eb01ab5b8_isrForCrosstalkSources_871267_38.stderr
parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: <AppFuture at 0x153d2b46b610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 38d8be64c9f970a926ef61718f0533d9
parsl.dataflow.memoization INFO: Task 45 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/ff806792-40f0-4fc0-a4e2-b72eec39e84c_isrForCrosstalkSources_871265_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/ff806792-40f0-4fc0-a4e2-b72eec39e84c_isrForCrosstalkSources_871265_27.stderr
parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: <AppFuture at 0x153d35aeb3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 27078ddda576c4b3d157311d5e71fbb3
parsl.dataflow.memoization INFO: Task 46 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/4c99f6d5-b4ea-4de4-ab09-d0e267fbbf5c_isrForCrosstalkSources_871267_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/4c99f6d5-b4ea-4de4-ab09-d0e267fbbf5c_isrForCrosstalkSources_871267_50.stderr
parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: <AppFuture at 0x153d35aebad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 2a467a54c6b6ca6c133473292eeb0dd2
parsl.dataflow.memoization INFO: Task 47 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8f67dd66-29f5-4a63-9d81-de067eee5b55_isrForCrosstalkSources_871265_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8f67dd66-29f5-4a63-9d81-de067eee5b55_isrForCrosstalkSources_871265_6.stderr
parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: <AppFuture at 0x153d35aea250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 67df0bbfdc62d958a379f6d5b8127f62
parsl.dataflow.memoization INFO: Task 48 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/9f4d259f-b2cd-4413-942b-b4c93fa7f9d4_isrForCrosstalkSources_871267_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/9f4d259f-b2cd-4413-942b-b4c93fa7f9d4_isrForCrosstalkSources_871267_52.stderr
parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: <AppFuture at 0x153d2b3d6c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 117d31e5ac9077a14d50e46b45c83c9d
parsl.dataflow.memoization INFO: Task 49 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/868f1835-8025-4b5a-b263-833133c847a4_isrForCrosstalkSources_871267_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/868f1835-8025-4b5a-b263-833133c847a4_isrForCrosstalkSources_871267_9.stderr
parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: <AppFuture at 0x153d2b3d5a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 50 has memoization hash e5035ba8f17b9ed86ae4e6f40835691b
parsl.dataflow.memoization INFO: Task 50 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b4f3c12b-94c3-4abb-9e32-c90c15b9b69b_isrForCrosstalkSources_871267_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b4f3c12b-94c3-4abb-9e32-c90c15b9b69b_isrForCrosstalkSources_871267_20.stderr
parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: <AppFuture at 0x153d2b6d2810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 123cffbbc0ad760f515c1e78949e7925
parsl.dataflow.memoization INFO: Task 51 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d74b10ac-5296-4207-9889-5e45d8dcce33_isrForCrosstalkSources_871267_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d74b10ac-5296-4207-9889-5e45d8dcce33_isrForCrosstalkSources_871267_34.stderr
parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: <AppFuture at 0x153d2b248990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 33d39b3ac2407edc6b248ec53dfba7b4
parsl.dataflow.memoization INFO: Task 52 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/23803679-ece2-482f-a196-942f5fe46322_isrForCrosstalkSources_871266_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/23803679-ece2-482f-a196-942f5fe46322_isrForCrosstalkSources_871266_39.stderr
parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: <AppFuture at 0x153d2b248450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 53 has memoization hash d2970c2ebf59d2bd0cb18142d9776f40
parsl.dataflow.memoization INFO: Task 53 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c6bc362d-a4e9-4ae0-9c1b-5ac94a7d7d72_isrForCrosstalkSources_871265_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c6bc362d-a4e9-4ae0-9c1b-5ac94a7d7d72_isrForCrosstalkSources_871265_39.stderr
parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: <AppFuture at 0x153d2b248e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 0cb3172c075c8e4278a78b5832235171
parsl.dataflow.memoization INFO: Task 54 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1ee7cc45-e718-45a3-9e6e-84d17fb37b87_isrForCrosstalkSources_871267_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1ee7cc45-e718-45a3-9e6e-84d17fb37b87_isrForCrosstalkSources_871267_15.stderr
parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: <AppFuture at 0x153d2b25b290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 93771cdb7b6476f6b38bc46849c36470
parsl.dataflow.memoization INFO: Task 55 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a4fb1ecb-351f-47cc-b98d-d5617b4bd01e_isrForCrosstalkSources_871266_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a4fb1ecb-351f-47cc-b98d-d5617b4bd01e_isrForCrosstalkSources_871266_7.stderr
parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: <AppFuture at 0x153d2b259790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 931e5fb738eff2f43c0d08973ecac321
parsl.dataflow.memoization INFO: Task 56 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/74de8ee1-4ff0-4b10-8b4a-202c69e9f60c_isrForCrosstalkSources_871267_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/74de8ee1-4ff0-4b10-8b4a-202c69e9f60c_isrForCrosstalkSources_871267_25.stderr
parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: <AppFuture at 0x153d2b6d2350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 57 has memoization hash d024663ea23feef2c6fa0902480d1b9b
parsl.dataflow.memoization INFO: Task 57 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/50d64024-d417-40e4-b0a8-6553b5736776_isrForCrosstalkSources_871267_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/50d64024-d417-40e4-b0a8-6553b5736776_isrForCrosstalkSources_871267_35.stderr
parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: <AppFuture at 0x153d2b25bd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 58 has memoization hash aae64dec77c40627bbe5603a2f4c4010
parsl.dataflow.memoization INFO: Task 58 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d09be141-2f83-4d12-897b-394b7a8d09ec_isrForCrosstalkSources_871267_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d09be141-2f83-4d12-897b-394b7a8d09ec_isrForCrosstalkSources_871267_61.stderr
parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: <AppFuture at 0x153d2b25bd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 1a69547b45bd35c37e4c1d1b666e266d
parsl.dataflow.memoization INFO: Task 59 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/dd62aa76-ff1a-480a-a239-11ff3722aa28_isrForCrosstalkSources_871267_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/dd62aa76-ff1a-480a-a239-11ff3722aa28_isrForCrosstalkSources_871267_3.stderr
parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: <AppFuture at 0x153d2b2dc8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 4f6930d8b095476bf7cd3d2fc5c4c766
parsl.dataflow.memoization INFO: Task 60 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/1e30023a-5e51-4bf9-809e-50d17b9e02e1_isrForCrosstalkSources_871266_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/1e30023a-5e51-4bf9-809e-50d17b9e02e1_isrForCrosstalkSources_871266_5.stderr
parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: <AppFuture at 0x153d2b2dcd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 61 has memoization hash e76a9b3ae780d09ece5faf72a7d0a6d5
parsl.dataflow.memoization INFO: Task 61 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/abba4e0e-fb71-4584-8707-366d42dece6a_isrForCrosstalkSources_871266_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/abba4e0e-fb71-4584-8707-366d42dece6a_isrForCrosstalkSources_871266_46.stderr
parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: <AppFuture at 0x153d2b2dc2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 7df1e1b154ad6f28506fada1a116fea9
parsl.dataflow.memoization INFO: Task 62 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/cb332051-af61-4d52-ad9d-ec442ba1dff2_isrForCrosstalkSources_871265_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/cb332051-af61-4d52-ad9d-ec442ba1dff2_isrForCrosstalkSources_871265_62.stderr
parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: <AppFuture at 0x153d2b2de010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 42d9adda434279992a6417303fad7030
parsl.dataflow.memoization INFO: Task 63 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c60f9697-eac3-4df6-9072-75a40d61e8d5_isrForCrosstalkSources_871265_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c60f9697-eac3-4df6-9072-75a40d61e8d5_isrForCrosstalkSources_871265_48.stderr
parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: <AppFuture at 0x153d2b2dde10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 220fedf9369336a40bdd42c1516e3e6e
parsl.dataflow.memoization INFO: Task 64 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/84de0381-9d55-4c1d-872c-a7d07f7523b1_isrForCrosstalkSources_871265_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/84de0381-9d55-4c1d-872c-a7d07f7523b1_isrForCrosstalkSources_871265_9.stderr
parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: <AppFuture at 0x153d2b2def50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 9e7ff7ed9e7a3859c4918ac4bf053350
parsl.dataflow.memoization INFO: Task 65 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/6faa79fd-6df2-4d48-ba84-8312ff56fe5d_isrForCrosstalkSources_871265_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/6faa79fd-6df2-4d48-ba84-8312ff56fe5d_isrForCrosstalkSources_871265_13.stderr
parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: <AppFuture at 0x153d2b2df950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 00d231f34d907aed82043c006a9524f3
parsl.dataflow.memoization INFO: Task 66 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e52ff88c-c359-4b21-a3f4-d40f2fa6317c_isrForCrosstalkSources_871267_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e52ff88c-c359-4b21-a3f4-d40f2fa6317c_isrForCrosstalkSources_871267_10.stderr
parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: <AppFuture at 0x153d2b258190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 977fb01a8725eed7ce1839d8df61c28a
parsl.dataflow.memoization INFO: Task 67 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0abdb050-5818-4978-82cf-8e923675fcfe_isrForCrosstalkSources_871267_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0abdb050-5818-4978-82cf-8e923675fcfe_isrForCrosstalkSources_871267_43.stderr
parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: <AppFuture at 0x153d2b2e1050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 68 has memoization hash dab541f64d142326874f4a2eee66acb9
parsl.dataflow.memoization INFO: Task 68 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/c6f0ca73-a56a-488e-82e8-90191a8d5a22_isrForCrosstalkSources_871266_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/c6f0ca73-a56a-488e-82e8-90191a8d5a22_isrForCrosstalkSources_871266_45.stderr
parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: <AppFuture at 0x153d2b2e21d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 69 has memoization hash d8237d50639392fe5afaeb57d21c2d9e
parsl.dataflow.memoization INFO: Task 69 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4fb10d2c-430c-4cdb-91dc-f2c121ef85da_isrForCrosstalkSources_871266_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4fb10d2c-430c-4cdb-91dc-f2c121ef85da_isrForCrosstalkSources_871266_55.stderr
parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: <AppFuture at 0x153d2b2e0bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 70 has memoization hash b59f81d999758cfa31802e88cf1797ce
parsl.dataflow.memoization INFO: Task 70 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b5f5da4f-4e44-44db-836e-d414b822c9c9_isrForCrosstalkSources_871266_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b5f5da4f-4e44-44db-836e-d414b822c9c9_isrForCrosstalkSources_871266_40.stderr
parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: <AppFuture at 0x153d2b2e20d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 9689d6bfb9c25769a2d4d8bb247e468d
parsl.dataflow.memoization INFO: Task 71 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/51497508-babe-4cff-8280-cd2b614676c4_isrForCrosstalkSources_871267_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/51497508-babe-4cff-8280-cd2b614676c4_isrForCrosstalkSources_871267_58.stderr
parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: <AppFuture at 0x153d2b2e33d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 72 has memoization hash d572a328716a522bf1335a40e00cdfe0
parsl.dataflow.memoization INFO: Task 72 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2a68200e-0e9a-4528-9858-ce447266e388_isrForCrosstalkSources_871265_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2a68200e-0e9a-4528-9858-ce447266e388_isrForCrosstalkSources_871265_20.stderr
parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: <AppFuture at 0x153d2b2e0210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 73 has memoization hash bc2e6d649ada1204a5822cf605763d2e
parsl.dataflow.memoization INFO: Task 73 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/69d6d115-61be-4d3b-8e10-74ba6e91aae1_isrForCrosstalkSources_871265_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/69d6d115-61be-4d3b-8e10-74ba6e91aae1_isrForCrosstalkSources_871265_37.stderr
parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: <AppFuture at 0x153d2b629190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 74 has memoization hash e5bcb53bc553b84662cc607f740f0f09
parsl.dataflow.memoization INFO: Task 74 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1c84e2a1-ce95-4f7f-99c8-6d85ef4fb2d7_isrForCrosstalkSources_871267_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1c84e2a1-ce95-4f7f-99c8-6d85ef4fb2d7_isrForCrosstalkSources_871267_30.stderr
parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: <AppFuture at 0x153d2b2f03d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 61b0fffa2a40f40e4416fb8264ea8045
parsl.dataflow.memoization INFO: Task 75 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/48016698-1f46-4b08-b219-099bdd5f6a83_isrForCrosstalkSources_871265_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/48016698-1f46-4b08-b219-099bdd5f6a83_isrForCrosstalkSources_871265_43.stderr
parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: <AppFuture at 0x153d2b2f1250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 860e00ac238adea0f37156cd5a9b43e2
parsl.dataflow.memoization INFO: Task 76 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e1c60f26-2d44-42fa-9708-853d828feece_isrForCrosstalkSources_871266_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e1c60f26-2d44-42fa-9708-853d828feece_isrForCrosstalkSources_871266_38.stderr
parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: <AppFuture at 0x153d2b2f1a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 77 has memoization hash dcf1e7fc225a6824b23c0dc9d81f0cb9
parsl.dataflow.memoization INFO: Task 77 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3bf203d7-5c0e-47df-aa86-ca11eda4d5d3_isrForCrosstalkSources_871265_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3bf203d7-5c0e-47df-aa86-ca11eda4d5d3_isrForCrosstalkSources_871265_10.stderr
parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: <AppFuture at 0x153d2b2f1d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 37356ac67243bef399819d6e4f8f0344
parsl.dataflow.memoization INFO: Task 78 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/258b8445-f8df-4cae-96d7-d58eaa8a9d65_isrForCrosstalkSources_871267_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/258b8445-f8df-4cae-96d7-d58eaa8a9d65_isrForCrosstalkSources_871267_46.stderr
parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: <AppFuture at 0x153d2b2f3a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 1d0f9db40ca71924994694e60bbce57c
parsl.dataflow.memoization INFO: Task 79 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/89b857d2-cc09-44b6-8449-2fddf717c90f_isrForCrosstalkSources_871265_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/89b857d2-cc09-44b6-8449-2fddf717c90f_isrForCrosstalkSources_871265_34.stderr
parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: <AppFuture at 0x153d2b2f3d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 80 has memoization hash d98a4873e9bd80e7444da0edff021f24
parsl.dataflow.memoization INFO: Task 80 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/91abcfc3-bf16-4307-ac74-ff97489cd88d_isrForCrosstalkSources_871265_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/91abcfc3-bf16-4307-ac74-ff97489cd88d_isrForCrosstalkSources_871265_33.stderr
parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: <AppFuture at 0x153d2b2f34d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 81 has memoization hash b1be86fd3ba1bde64f9dcee84c69ea1d
parsl.dataflow.memoization INFO: Task 81 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8e4afc23-f3f4-4d67-a78a-72b095e89190_isrForCrosstalkSources_871265_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8e4afc23-f3f4-4d67-a78a-72b095e89190_isrForCrosstalkSources_871265_29.stderr
parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: <AppFuture at 0x153d2b2dd250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 1f1cb38932ebfccc7e4b8b5b06d90722
parsl.dataflow.memoization INFO: Task 82 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/537bca6c-87eb-42cc-ab2e-269a30ecd077_isrForCrosstalkSources_871265_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/537bca6c-87eb-42cc-ab2e-269a30ecd077_isrForCrosstalkSources_871265_21.stderr
parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: <AppFuture at 0x153d2b2fd4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 7a7f2a5501b10c4c8984c34e72d6a7b4
parsl.dataflow.memoization INFO: Task 83 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5563fe49-11b4-4c38-a4ee-d7b5b7d4551d_isrForCrosstalkSources_871266_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5563fe49-11b4-4c38-a4ee-d7b5b7d4551d_isrForCrosstalkSources_871266_24.stderr
parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: <AppFuture at 0x153d2b2fd310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 0dc6afc89e20022c1b79b7baa285a6c8
parsl.dataflow.memoization INFO: Task 84 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/af087b70-5d90-4da2-b23d-f93117a4cdbf_isrForCrosstalkSources_871266_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/af087b70-5d90-4da2-b23d-f93117a4cdbf_isrForCrosstalkSources_871266_28.stderr
parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: <AppFuture at 0x153d2b2fe590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 85 has memoization hash a0731eb708c7bbf37130a338e6c4cacf
parsl.dataflow.memoization INFO: Task 85 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5244671d-1dab-4841-a1f5-306ee06f68df_isrForCrosstalkSources_871265_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5244671d-1dab-4841-a1f5-306ee06f68df_isrForCrosstalkSources_871265_32.stderr
parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: <AppFuture at 0x153d2b2ff310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 86 has memoization hash a2e4b76bcec03536646632a68d93b8b7
parsl.dataflow.memoization INFO: Task 86 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/1f6fbe36-5dd0-428f-9b56-58157fc7027f_isrForCrosstalkSources_871265_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/1f6fbe36-5dd0-428f-9b56-58157fc7027f_isrForCrosstalkSources_871265_61.stderr
parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: <AppFuture at 0x153d2b2ffad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 87 has memoization hash eab85e04f0d9be022e6a4827ce2eeac4
parsl.dataflow.memoization INFO: Task 87 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/d9386eb4-c10c-4840-bd3a-5034434e98cb_isrForCrosstalkSources_871266_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/d9386eb4-c10c-4840-bd3a-5034434e98cb_isrForCrosstalkSources_871266_33.stderr
parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: <AppFuture at 0x153d2b2ffcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 88 has memoization hash fe49d77c0adc73289d1631ce9538d8a1
parsl.dataflow.memoization INFO: Task 88 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/5f98a1bd-3726-4bd1-9b14-3aeffedee265_isrForCrosstalkSources_871267_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/5f98a1bd-3726-4bd1-9b14-3aeffedee265_isrForCrosstalkSources_871267_6.stderr
parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: <AppFuture at 0x153d2b2e0d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 89 has memoization hash a842f4b5c66d68f4a8192a52dcce0eab
parsl.dataflow.memoization INFO: Task 89 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4a0af287-a0ae-455c-83d4-f1943eb61c15_isrForCrosstalkSources_871265_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4a0af287-a0ae-455c-83d4-f1943eb61c15_isrForCrosstalkSources_871265_1.stderr
parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: <AppFuture at 0x153d2b308c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 90 has memoization hash aee41afffaf9869a1f15603498cf474b
parsl.dataflow.memoization INFO: Task 90 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/c9f023f1-6b76-46b6-9eea-d95013c7bf2a_isrForCrosstalkSources_871267_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/c9f023f1-6b76-46b6-9eea-d95013c7bf2a_isrForCrosstalkSources_871267_8.stderr
parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: <AppFuture at 0x153d2b3099d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 3f3bd2ace79a9ca2211f5d1deac7c43b
parsl.dataflow.memoization INFO: Task 91 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/13b38b74-0308-498d-b47d-028b595e78a0_isrForCrosstalkSources_871265_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/13b38b74-0308-498d-b47d-028b595e78a0_isrForCrosstalkSources_871265_17.stderr
parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: <AppFuture at 0x153d2b30a310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 224f20f2cf426326986344c621a0a391
parsl.dataflow.memoization INFO: Task 92 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/cf1506f4-9d6b-4959-a598-eac156c6ea16_isrForCrosstalkSources_871267_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/cf1506f4-9d6b-4959-a598-eac156c6ea16_isrForCrosstalkSources_871267_48.stderr
parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: <AppFuture at 0x153d2b30ba10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 436cdec4ebcadd340c17369feb4378dd
parsl.dataflow.memoization INFO: Task 93 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e054228d-a6ad-444d-b039-b185d5baf072_isrForCrosstalkSources_871266_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e054228d-a6ad-444d-b039-b185d5baf072_isrForCrosstalkSources_871266_49.stderr
parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: <AppFuture at 0x153d2b30bd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 94 has memoization hash dbdbc0289730be458d93a0d88c28c029
parsl.dataflow.memoization INFO: Task 94 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a1dfe69e-9ca0-432b-92b1-bddc45460f94_isrForCrosstalkSources_871266_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a1dfe69e-9ca0-432b-92b1-bddc45460f94_isrForCrosstalkSources_871266_8.stderr
parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: <AppFuture at 0x153d2b309fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 27835a72abd0daab7fbd86a79bec94a9
parsl.dataflow.memoization INFO: Task 95 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0992a95f-40a0-45b2-aed5-a0d79c12ad85_isrForCrosstalkSources_871267_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0992a95f-40a0-45b2-aed5-a0d79c12ad85_isrForCrosstalkSources_871267_53.stderr
parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: <AppFuture at 0x153d2b314a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 0497981527dc88839fc13cfe0927386a
parsl.dataflow.memoization INFO: Task 96 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2f54e10e-a39c-495b-91f4-b8a0b40c043f_isrForCrosstalkSources_871265_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2f54e10e-a39c-495b-91f4-b8a0b40c043f_isrForCrosstalkSources_871265_7.stderr
parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: <AppFuture at 0x153d2b315d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 97 has memoization hash d701238f27ddd8abac64a8cab980dee0
parsl.dataflow.memoization INFO: Task 97 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1086f740-5b5e-48e6-b279-5db5205addee_isrForCrosstalkSources_871267_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1086f740-5b5e-48e6-b279-5db5205addee_isrForCrosstalkSources_871267_60.stderr
parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: <AppFuture at 0x153d2b315ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 98 has memoization hash a25062da52c911555adf48039a0f35e4
parsl.dataflow.memoization INFO: Task 98 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2ab9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a7cd218a-447a-4a30-aaa6-af20a23f697a_isrForCrosstalkSources_871266_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a7cd218a-447a-4a30-aaa6-af20a23f697a_isrForCrosstalkSources_871266_9.stderr
parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: <AppFuture at 0x153d2b3155d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 6cda6836ed68b0a876f248a46a6f158e
parsl.dataflow.memoization INFO: Task 99 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2aba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5cebd64f-e7a9-425f-b919-8509f5b2dc15_isrForCrosstalkSources_871266_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5cebd64f-e7a9-425f-b919-8509f5b2dc15_isrForCrosstalkSources_871266_53.stderr
parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: <AppFuture at 0x153d2b316810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 2c8cf9b3b2f2b2ee68df5ccf3a52e58f
parsl.dataflow.memoization INFO: Task 100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2abb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b0a70a19-6f24-4e99-aa32-8bed22246631_isrForCrosstalkSources_871265_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b0a70a19-6f24-4e99-aa32-8bed22246631_isrForCrosstalkSources_871265_38.stderr
parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: <AppFuture at 0x153d2b3178d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 423b1ac4b0be2a8de390d5e2d3c5736c
parsl.dataflow.memoization INFO: Task 101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2abc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/6640bd41-f59d-4570-81c4-67dd500a2b92_isrForCrosstalkSources_871265_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/6640bd41-f59d-4570-81c4-67dd500a2b92_isrForCrosstalkSources_871265_26.stderr
parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: <AppFuture at 0x153d2b317fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 05506e0cf369372abaf19482e7f928be
parsl.dataflow.memoization INFO: Task 102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2abba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/455a6be3-3752-4ab9-b264-406a2fdb5554_isrForCrosstalkSources_871266_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/455a6be3-3752-4ab9-b264-406a2fdb5554_isrForCrosstalkSources_871266_51.stderr
parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: <AppFuture at 0x153d2b31c910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 103 has memoization hash c2a421f0e3f4ce51279d12896f026f16
parsl.dataflow.memoization INFO: Task 103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2abce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/bbb65d4e-0779-49c1-b30a-34ac1e1a40f8_isrForCrosstalkSources_871266_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/bbb65d4e-0779-49c1-b30a-34ac1e1a40f8_isrForCrosstalkSources_871266_43.stderr
parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: <AppFuture at 0x153d2b31de90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 104 has memoization hash ccbe1cb6adc971249984912966413cd8
parsl.dataflow.memoization INFO: Task 104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2abd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b7c8b0a8-b2fd-48bf-994d-d965654a7f0b_isrForCrosstalkSources_871267_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b7c8b0a8-b2fd-48bf-994d-d965654a7f0b_isrForCrosstalkSources_871267_41.stderr
parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: <AppFuture at 0x153d2b31dad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 30459d3d64aa3cfc5f8bd0a6cb71479a
parsl.dataflow.memoization INFO: Task 105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2abe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/db606f06-2680-41cc-a8aa-0f842d567698_isrForCrosstalkSources_871267_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/db606f06-2680-41cc-a8aa-0f842d567698_isrForCrosstalkSources_871267_32.stderr
parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: <AppFuture at 0x153d2b31e250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 79f6cced68af980fbf43d58342cac731
parsl.dataflow.memoization INFO: Task 106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2abf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b466e922-a669-4734-be94-34b7403b3b66_isrForCrosstalkSources_871267_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b466e922-a669-4734-be94-34b7403b3b66_isrForCrosstalkSources_871267_22.stderr
parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: <AppFuture at 0x153d2b31ec10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 210262d3c45a50b63d8209ec18fca6f1
parsl.dataflow.memoization INFO: Task 107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b2a8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/931d22fc-d736-420f-8198-bc23fcf9269a_isrForCrosstalkSources_871267_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/931d22fc-d736-420f-8198-bc23fcf9269a_isrForCrosstalkSources_871267_1.stderr
parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: <AppFuture at 0x153d2b31e650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 8322409ac54bddf6e1ed2f16ab9fe0a3
parsl.dataflow.memoization INFO: Task 108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3240e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5bb08627-b89e-4f67-a7f2-d30e838605e8_isrForCrosstalkSources_871265_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5bb08627-b89e-4f67-a7f2-d30e838605e8_isrForCrosstalkSources_871265_12.stderr
parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: <AppFuture at 0x153d2b31f910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 024e0ae133d195630be50aca91c2b5af
parsl.dataflow.memoization INFO: Task 109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/1cbb62d2-c1d8-4a74-be42-a83d85233590_isrForCrosstalkSources_871266_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/1cbb62d2-c1d8-4a74-be42-a83d85233590_isrForCrosstalkSources_871266_2.stderr
parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: <AppFuture at 0x153d2b32ce10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 809267a51ada5d2bf4dffc5cacc603c6
parsl.dataflow.memoization INFO: Task 110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3242c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/27df0f72-8176-4736-b2c9-fb3fd82f6d82_isrForCrosstalkSources_871266_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/27df0f72-8176-4736-b2c9-fb3fd82f6d82_isrForCrosstalkSources_871266_14.stderr
parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: <AppFuture at 0x153d2b32c950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 4a448d147f69f13bb227d760edefb249
parsl.dataflow.memoization INFO: Task 111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/155404bd-a88b-46b3-bc6b-14b4f649b3a0_isrForCrosstalkSources_871266_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/155404bd-a88b-46b3-bc6b-14b4f649b3a0_isrForCrosstalkSources_871266_27.stderr
parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: <AppFuture at 0x153d2b32d910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 7e178f268c4e88728fb58e7da7f1ddf1
parsl.dataflow.memoization INFO: Task 112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/d4e154f3-c05d-45fb-878d-f8c320c11935_isrForCrosstalkSources_871266_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/d4e154f3-c05d-45fb-878d-f8c320c11935_isrForCrosstalkSources_871266_30.stderr
parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: <AppFuture at 0x153d2b32e250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 113 has memoization hash c5ab54a16ea6a4221864a8a0ddd19d8d
parsl.dataflow.memoization INFO: Task 113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/94576fc4-e55b-4127-9a5e-31bd1448dbeb_isrForCrosstalkSources_871265_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/94576fc4-e55b-4127-9a5e-31bd1448dbeb_isrForCrosstalkSources_871265_54.stderr
parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: <AppFuture at 0x153d2b32ea10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 4544792b43d900361153f754fccfd704
parsl.dataflow.memoization INFO: Task 114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3244a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/92c4d2a1-9a63-4c2b-99fa-b79be9dcd7cc_isrForCrosstalkSources_871266_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/92c4d2a1-9a63-4c2b-99fa-b79be9dcd7cc_isrForCrosstalkSources_871266_47.stderr
parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: <AppFuture at 0x153d2b32fe50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 115 has memoization hash b529079a8e2eaac2855400c6b19bb382
parsl.dataflow.memoization INFO: Task 115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3245e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/7278cb21-00ac-4168-a1ff-da2599fba5ee_isrForCrosstalkSources_871267_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/7278cb21-00ac-4168-a1ff-da2599fba5ee_isrForCrosstalkSources_871267_29.stderr
parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: <AppFuture at 0x153d2b32fb50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 400debad5c38bae28b31303d3f9f21dc
parsl.dataflow.memoization INFO: Task 116 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/8b3632c6-4d5a-44e3-9862-20a0dd829c13_isrForCrosstalkSources_871266_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/8b3632c6-4d5a-44e3-9862-20a0dd829c13_isrForCrosstalkSources_871266_35.stderr
parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: <AppFuture at 0x153d2b339350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 8ab006a2839ecedef33bb70d0c9e8265
parsl.dataflow.memoization INFO: Task 117 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/d2cbcc0e-e43f-4d9d-a4ce-7985b6b39032_isrForCrosstalkSources_871265_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/d2cbcc0e-e43f-4d9d-a4ce-7985b6b39032_isrForCrosstalkSources_871265_24.stderr
parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: <AppFuture at 0x153d2b338610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 92399324e78471a8eb65489d11328d25
parsl.dataflow.memoization INFO: Task 118 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b09c8eb4-9399-410e-81c0-619e035a9eb3_isrForCrosstalkSources_871267_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b09c8eb4-9399-410e-81c0-619e035a9eb3_isrForCrosstalkSources_871267_12.stderr
parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: <AppFuture at 0x153d2b339590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 74a853701ebac6729706af89167f1645
parsl.dataflow.memoization INFO: Task 119 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3247c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fe4a9b68-9b82-492a-b91e-eb9f742d9fbf_isrForCrosstalkSources_871267_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fe4a9b68-9b82-492a-b91e-eb9f742d9fbf_isrForCrosstalkSources_871267_42.stderr
parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: <AppFuture at 0x153d2b33a250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 01ace307b47ccd14deddc53fe11a752e
parsl.dataflow.memoization INFO: Task 120 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/149e5493-d788-47b6-8375-ced13f896f9d_isrForCrosstalkSources_871266_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/149e5493-d788-47b6-8375-ced13f896f9d_isrForCrosstalkSources_871266_34.stderr
parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: <AppFuture at 0x153d2b33a950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 53f346ce080281a0f49bfef3238f8e68
parsl.dataflow.memoization INFO: Task 121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/199f01e9-e3a3-4d43-95b0-d404cca3c734_isrForCrosstalkSources_871266_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/199f01e9-e3a3-4d43-95b0-d404cca3c734_isrForCrosstalkSources_871266_22.stderr
parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: <AppFuture at 0x153d2b33b7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 122 has memoization hash e9bea8061bc34f37d825d597a851c541
parsl.dataflow.memoization INFO: Task 122 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3249a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/f270b49f-7959-4091-b93c-4115f518ce91_isrForCrosstalkSources_871267_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/f270b49f-7959-4091-b93c-4115f518ce91_isrForCrosstalkSources_871267_49.stderr
parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: <AppFuture at 0x153d2b33bc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 123 has memoization hash b2e3fc057dfb69b96940b8fae322d027
parsl.dataflow.memoization INFO: Task 123 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/947d9bcc-9c3d-49f9-a6fe-a556b9cbbb6b_isrForCrosstalkSources_871266_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/947d9bcc-9c3d-49f9-a6fe-a556b9cbbb6b_isrForCrosstalkSources_871266_61.stderr
parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: <AppFuture at 0x153d29d1c2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 473ef45ca01a37c841347eeda5d4304f
parsl.dataflow.memoization INFO: Task 124 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/86832478-9781-45e2-b3ac-b73a24e5a627_isrForCrosstalkSources_871265_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/86832478-9781-45e2-b3ac-b73a24e5a627_isrForCrosstalkSources_871265_28.stderr
parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: <AppFuture at 0x153d29d1d510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 3b8ffd35285c3288fb2f7a8139397b3b
parsl.dataflow.memoization INFO: Task 125 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4b9b0c85-027c-4fe4-ab86-e271f7a76429_isrForCrosstalkSources_871265_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4b9b0c85-027c-4fe4-ab86-e271f7a76429_isrForCrosstalkSources_871265_19.stderr
parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: <AppFuture at 0x153d29d1d950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 33d2e886da381e874efaf3d569f76ac1
parsl.dataflow.memoization INFO: Task 126 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/da535170-cfd0-4c79-8566-46091f589f65_isrForCrosstalkSources_871265_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/da535170-cfd0-4c79-8566-46091f589f65_isrForCrosstalkSources_871265_51.stderr
parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: <AppFuture at 0x153d29d1e710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 555312106a5afac92508c0476debeed1
parsl.dataflow.memoization INFO: Task 127 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/14d97276-fd43-4f46-96de-d837fa45bd35_isrForCrosstalkSources_871267_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/14d97276-fd43-4f46-96de-d837fa45bd35_isrForCrosstalkSources_871267_19.stderr
parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: <AppFuture at 0x153d29d1e790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 98b50018e928fef090d468931be24da7
parsl.dataflow.memoization INFO: Task 128 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f09f8b52-d20d-4f04-81cf-6a6e08f8050e_isrForCrosstalkSources_871266_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f09f8b52-d20d-4f04-81cf-6a6e08f8050e_isrForCrosstalkSources_871266_6.stderr
parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: <AppFuture at 0x153d29d1f290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 129 has memoization hash eab1f30a8f632e7e034e3b93585847db
parsl.dataflow.memoization INFO: Task 129 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/15f0de64-dfec-4398-ab35-7c05878762a7_isrForCrosstalkSources_871265_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/15f0de64-dfec-4398-ab35-7c05878762a7_isrForCrosstalkSources_871265_53.stderr
parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: <AppFuture at 0x153d29d1fad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 9d89bdc573634f57623c01b15628f10b
parsl.dataflow.memoization INFO: Task 130 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3ab41666-d0a8-426a-8b91-b484f2bf120f_isrForCrosstalkSources_871265_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3ab41666-d0a8-426a-8b91-b484f2bf120f_isrForCrosstalkSources_871265_58.stderr
parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: <AppFuture at 0x153d29d1f690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 3c26fdb82b7d568175af909dbaf78baf
parsl.dataflow.memoization INFO: Task 131 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f658b290-bc51-46f7-b6ad-ad6ce3437f7e_isrForCrosstalkSources_871266_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f658b290-bc51-46f7-b6ad-ad6ce3437f7e_isrForCrosstalkSources_871266_60.stderr
parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: <AppFuture at 0x153d29d24d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 65a14a6df5ccfd520c8d9a76a7a01ced
parsl.dataflow.memoization INFO: Task 132 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b324fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b0447118-d1a6-4398-ac91-6b9c10575e8b_isrForCrosstalkSources_871266_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b0447118-d1a6-4398-ac91-6b9c10575e8b_isrForCrosstalkSources_871266_20.stderr
parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: <AppFuture at 0x153d29d25410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 94c12af85bdcc071558cfa83207426e2
parsl.dataflow.memoization INFO: Task 133 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/072c1416-99f3-4b41-b468-ac9f27454d72_isrForCrosstalkSources_871267_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/072c1416-99f3-4b41-b468-ac9f27454d72_isrForCrosstalkSources_871267_37.stderr
parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: <AppFuture at 0x153d29d24f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 2b838fc2ab38f7d4ee1bca50f4cb9d3a
parsl.dataflow.memoization INFO: Task 134 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/42c2145e-fc23-4573-8e26-8820d7a97fd5_isrForCrosstalkSources_871265_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/42c2145e-fc23-4573-8e26-8820d7a97fd5_isrForCrosstalkSources_871265_30.stderr
parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: <AppFuture at 0x153d29d263d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 5ef537479a7885f4b0d6aae6d236197b
parsl.dataflow.memoization INFO: Task 135 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/de65e774-d6a4-4753-acb6-6c3b5f75d0b1_isrForCrosstalkSources_871265_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/de65e774-d6a4-4753-acb6-6c3b5f75d0b1_isrForCrosstalkSources_871265_40.stderr
parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: <AppFuture at 0x153d29d265d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 523ec6b87be93e60cfe0ec646a1bf366
parsl.dataflow.memoization INFO: Task 136 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3251c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e6f8e3f2-1508-4300-97e6-8adefc752d48_isrForCrosstalkSources_871266_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e6f8e3f2-1508-4300-97e6-8adefc752d48_isrForCrosstalkSources_871266_12.stderr
parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: <AppFuture at 0x153d29d26810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 137 has memoization hash ec8dea4ecdafa55123b7b5b928836d92
parsl.dataflow.memoization INFO: Task 137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/a23a44c0-e989-445b-bc6d-1276d87aa882_isrForCrosstalkSources_871265_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/a23a44c0-e989-445b-bc6d-1276d87aa882_isrForCrosstalkSources_871265_57.stderr
parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: <AppFuture at 0x153d2b3142d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 138 has memoization hash da957c7c5e2d13a7e6f27ff4b81b5f97
parsl.dataflow.memoization INFO: Task 138 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3253a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/1758d504-2afc-4a89-8a2d-8466e63f5f39_isrForCrosstalkSources_871265_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/1758d504-2afc-4a89-8a2d-8466e63f5f39_isrForCrosstalkSources_871265_25.stderr
parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: <AppFuture at 0x153d29d2c650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 139 has memoization hash f74116eb2bc7fddbaad35d821cb1b476
parsl.dataflow.memoization INFO: Task 139 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b771577c-2494-440b-adc2-267a42afc903_isrForCrosstalkSources_871267_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b771577c-2494-440b-adc2-267a42afc903_isrForCrosstalkSources_871267_16.stderr
parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: <AppFuture at 0x153d29d2ded0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 140 has memoization hash f2f8b49dfa4a690353a7b1595373f329
parsl.dataflow.memoization INFO: Task 140 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/dfe32577-59b4-4a21-9d1a-a0d5ac7faa4b_isrForCrosstalkSources_871266_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/dfe32577-59b4-4a21-9d1a-a0d5ac7faa4b_isrForCrosstalkSources_871266_15.stderr
parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: <AppFuture at 0x153d29d2dd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 2a3c00669c66761381e38598792b9b51
parsl.dataflow.memoization INFO: Task 141 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3254e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2e1470f6-62e3-4281-a8d8-9b8e941dbf79_isrForCrosstalkSources_871265_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2e1470f6-62e3-4281-a8d8-9b8e941dbf79_isrForCrosstalkSources_871265_36.stderr
parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: <AppFuture at 0x153d29d2d950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 142 has memoization hash f846a4742c43381ac2848dceac8f1f35
parsl.dataflow.memoization INFO: Task 142 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/45c93c2d-f179-48a6-b033-1ee3e0bf504b_isrForCrosstalkSources_871265_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/45c93c2d-f179-48a6-b033-1ee3e0bf504b_isrForCrosstalkSources_871265_41.stderr
parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: <AppFuture at 0x153d29d2f650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 143 has memoization hash b456a90809b62182c61f139587550dc3
parsl.dataflow.memoization INFO: Task 143 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3256c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/eb15782f-61ba-4110-bca3-70b6168b62d4_isrForCrosstalkSources_871266_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/eb15782f-61ba-4110-bca3-70b6168b62d4_isrForCrosstalkSources_871266_19.stderr
parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: <AppFuture at 0x153d29d2ec50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 4de911fbb647da0f0ce9459eb98cdb8d
parsl.dataflow.memoization INFO: Task 144 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c1cf8330-9811-4146-9c13-ffe010a2952f_isrForCrosstalkSources_871265_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c1cf8330-9811-4146-9c13-ffe010a2952f_isrForCrosstalkSources_871265_2.stderr
parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: <AppFuture at 0x153d29d2e790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 6f7c5208a94f6b1959a360f27deea2dc
parsl.dataflow.memoization INFO: Task 145 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3258a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a082ef0f-3891-4c42-979f-697c1a22035d_isrForCrosstalkSources_871266_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a082ef0f-3891-4c42-979f-697c1a22035d_isrForCrosstalkSources_871266_13.stderr
parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: <AppFuture at 0x153d29d1c090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 146 has memoization hash d9c712debb1a5daaa84cc538bd3b99e5
parsl.dataflow.memoization INFO: Task 146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0b47a4c4-8c4a-4935-bb13-63a1ccb67bed_isrForCrosstalkSources_871267_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0b47a4c4-8c4a-4935-bb13-63a1ccb67bed_isrForCrosstalkSources_871267_7.stderr
parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: <AppFuture at 0x153d29d35510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 52a0d1f239289edfa5974d06c2dc7543
parsl.dataflow.memoization INFO: Task 147 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/73fd5ed7-b7e3-4ead-86bf-181e87f98215_isrForCrosstalkSources_871265_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/73fd5ed7-b7e3-4ead-86bf-181e87f98215_isrForCrosstalkSources_871265_52.stderr
parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: <AppFuture at 0x153d29d2e150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 148 has memoization hash f9d0208c8929584eb6b6f28fd9e4c702
parsl.dataflow.memoization INFO: Task 148 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3259e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/ee17da96-dc19-4683-ab5a-f55c753d5345_isrForCrosstalkSources_871266_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/ee17da96-dc19-4683-ab5a-f55c753d5345_isrForCrosstalkSources_871266_52.stderr
parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: <AppFuture at 0x153d29d35e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 149 has memoization hash d5d01bfa115eb89af0b73383153bfea8
parsl.dataflow.memoization INFO: Task 149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0b958f18-afae-47c0-b1b6-705c31f66395_isrForCrosstalkSources_871267_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0b958f18-afae-47c0-b1b6-705c31f66395_isrForCrosstalkSources_871267_40.stderr
parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: <AppFuture at 0x153d29d36990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 8efa3f046e2df99875a48ea77aeb8dee
parsl.dataflow.memoization INFO: Task 150 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f937b6b8-7134-4946-85b4-f827b58ce531_isrForCrosstalkSources_871266_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f937b6b8-7134-4946-85b4-f827b58ce531_isrForCrosstalkSources_871266_3.stderr
parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: <AppFuture at 0x153d29d36e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 9d7ae4b9a5e794200d025032d1beaa07
parsl.dataflow.memoization INFO: Task 151 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e2580183-d4aa-4ae7-92fb-83f24713822e_isrForCrosstalkSources_871267_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e2580183-d4aa-4ae7-92fb-83f24713822e_isrForCrosstalkSources_871267_13.stderr
parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: <AppFuture at 0x153d29d37390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 685b1e338bc80bd7925ee06be1a19867
parsl.dataflow.memoization INFO: Task 152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/387c8d11-bc9f-40d0-a605-1b500ea0fb19_isrForCrosstalkSources_871266_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/387c8d11-bc9f-40d0-a605-1b500ea0fb19_isrForCrosstalkSources_871266_4.stderr
parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: <AppFuture at 0x153d29d34250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 153 has memoization hash a54b144860c34ff87a840fad081c0c7a
parsl.dataflow.memoization INFO: Task 153 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b46d5046-37bb-43d7-a18a-48b62f763ec3_isrForCrosstalkSources_871265_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b46d5046-37bb-43d7-a18a-48b62f763ec3_isrForCrosstalkSources_871265_31.stderr
parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: <AppFuture at 0x153d2b32ce50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 154 has memoization hash bddd4eb0fa716f1d283ddad9ef1cd22f
parsl.dataflow.memoization INFO: Task 154 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/9e865543-9449-482b-96ac-2edbf6c24c2a_isrForCrosstalkSources_871265_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/9e865543-9449-482b-96ac-2edbf6c24c2a_isrForCrosstalkSources_871265_47.stderr
parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: <AppFuture at 0x153d29d3c610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 155 has memoization hash f3fe79dcc394c5d6337849e06a02ed5c
parsl.dataflow.memoization INFO: Task 155 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/6e37e05d-9944-43c7-96c1-4bbdb6f72f5b_isrForCrosstalkSources_871267_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/6e37e05d-9944-43c7-96c1-4bbdb6f72f5b_isrForCrosstalkSources_871267_14.stderr
parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: <AppFuture at 0x153d29d3d4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 156 has memoization hash c6cdd67396c4e279747b7f5b7622a826
parsl.dataflow.memoization INFO: Task 156 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/02731c8a-de60-44b7-9b00-9fc84e62c3f8_isrForCrosstalkSources_871267_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/02731c8a-de60-44b7-9b00-9fc84e62c3f8_isrForCrosstalkSources_871267_51.stderr
parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: <AppFuture at 0x153d2b31efd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 9dceac1c028b53d7f7665d5c6a1d570b
parsl.dataflow.memoization INFO: Task 157 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b325f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/af734154-783b-46ce-83d8-a77f09309118_isrForCrosstalkSources_871267_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/af734154-783b-46ce-83d8-a77f09309118_isrForCrosstalkSources_871267_2.stderr
parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: <AppFuture at 0x153d29d3eb90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 158 has memoization hash a03c94268d033d6c4075c976f920c823
parsl.dataflow.memoization INFO: Task 158 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/15c84f6b-371b-4061-a2eb-f812013a34a6_isrForCrosstalkSources_871267_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/15c84f6b-371b-4061-a2eb-f812013a34a6_isrForCrosstalkSources_871267_31.stderr
parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: <AppFuture at 0x153d29d3f3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 13ae4fd2888a32d123b8f7bb6a775a61
parsl.dataflow.memoization INFO: Task 159 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3260c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/4bd0a408-641f-44bc-ba28-2b88e75b43c5_isrForCrosstalkSources_871267_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/4bd0a408-641f-44bc-ba28-2b88e75b43c5_isrForCrosstalkSources_871267_44.stderr
parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: <AppFuture at 0x153d29d3f4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 160 has memoization hash b0f0600d0429029b5b2fd2f65c569ab1
parsl.dataflow.memoization INFO: Task 160 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5ab12821-ce48-4d51-b618-94f44ef69b1a_isrForCrosstalkSources_871265_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5ab12821-ce48-4d51-b618-94f44ef69b1a_isrForCrosstalkSources_871265_50.stderr
parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: <AppFuture at 0x153d2b338390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 95ec47f38bf048bf84314f4158de4a81
parsl.dataflow.memoization INFO: Task 161 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/84ceeafb-0d07-4be5-a387-9628ae907c6f_isrForCrosstalkSources_871265_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/84ceeafb-0d07-4be5-a387-9628ae907c6f_isrForCrosstalkSources_871265_3.stderr
parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: <AppFuture at 0x153d29d50c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 6d62ec1e791c48ca836938113ac89a83
parsl.dataflow.memoization INFO: Task 162 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3262a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3745997b-9d12-4531-9a59-d8c164af8626_isrForCrosstalkSources_871265_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3745997b-9d12-4531-9a59-d8c164af8626_isrForCrosstalkSources_871265_60.stderr
parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: <AppFuture at 0x153d29d50fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 38864e0cf8346dffa1a6cd27020bce84
parsl.dataflow.memoization INFO: Task 163 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/50eb28c2-eb1f-435b-9dee-45bd60d072d6_isrForCrosstalkSources_871267_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/50eb28c2-eb1f-435b-9dee-45bd60d072d6_isrForCrosstalkSources_871267_45.stderr
parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: <AppFuture at 0x153d29d51310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 230e21e33029c9f509f76732b0d1cec2
parsl.dataflow.memoization INFO: Task 164 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3263e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/21b38c98-1e75-4644-bcba-50b5710fc99c_isrForCrosstalkSources_871266_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/21b38c98-1e75-4644-bcba-50b5710fc99c_isrForCrosstalkSources_871266_32.stderr
parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: <AppFuture at 0x153d29d52690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 165 has memoization hash a9b8eabf3b90788e91543780f970fda1
parsl.dataflow.memoization INFO: Task 165 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5b78c897-7a82-468b-ae11-3f015ef0c559_isrForCrosstalkSources_871266_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5b78c897-7a82-468b-ae11-3f015ef0c559_isrForCrosstalkSources_871266_48.stderr
parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: <AppFuture at 0x153d29d53510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 78754eb925714e7e8e94c63634dc7c56
parsl.dataflow.memoization INFO: Task 166 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/20fef37a-5fee-4607-a02c-86f3bff08d8a_isrForCrosstalkSources_871266_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/20fef37a-5fee-4607-a02c-86f3bff08d8a_isrForCrosstalkSources_871266_37.stderr
parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: <AppFuture at 0x153d29d53d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 3008b1515b187097ac2667b30c3d3ecf
parsl.dataflow.memoization INFO: Task 167 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3265c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/42e51c68-307c-415d-b4c5-f8e3378e18d1_isrForCrosstalkSources_871267_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/42e51c68-307c-415d-b4c5-f8e3378e18d1_isrForCrosstalkSources_871267_21.stderr
parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: <AppFuture at 0x153d2b30afd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 168 has memoization hash afc993c3a50b33838b7df577521e792d
parsl.dataflow.memoization INFO: Task 168 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/7f6b9fae-5d71-4ca3-b1aa-2dc0872c2a9a_isrForCrosstalkSources_871266_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/7f6b9fae-5d71-4ca3-b1aa-2dc0872c2a9a_isrForCrosstalkSources_871266_59.stderr
parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: <AppFuture at 0x153d29d585d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 16bbd2ac4b626a4eacd50f9b2ed877ba
parsl.dataflow.memoization INFO: Task 169 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/eb5bd190-f164-4e22-8921-9d8e87afaa89_isrForCrosstalkSources_871265_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/eb5bd190-f164-4e22-8921-9d8e87afaa89_isrForCrosstalkSources_871265_11.stderr
parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: <AppFuture at 0x153d29d53410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 802a4a27b72d4cf2ebe52c8b9c8642c9
parsl.dataflow.memoization INFO: Task 170 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/09289095-185b-4e1a-8509-4755fa9b1506_isrForCrosstalkSources_871265_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/09289095-185b-4e1a-8509-4755fa9b1506_isrForCrosstalkSources_871265_8.stderr
parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: <AppFuture at 0x153d29d5a510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 171 has memoization hash b1d54d907ec7b70a6c03c7a6703b9c2c
parsl.dataflow.memoization INFO: Task 171 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3267a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/fb76c9a1-7128-4319-99fc-f5fb2e9a08d2_isrForCrosstalkSources_871266_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/fb76c9a1-7128-4319-99fc-f5fb2e9a08d2_isrForCrosstalkSources_871266_41.stderr
parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: <AppFuture at 0x153d29d5a4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 172 has memoization hash b13e6caa89149567b90da8243cda7190
parsl.dataflow.memoization INFO: Task 172 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3268e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/ede6572d-50c8-4a21-9733-4ad274289c9a_isrForCrosstalkSources_871266_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/ede6572d-50c8-4a21-9733-4ad274289c9a_isrForCrosstalkSources_871266_26.stderr
parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: <AppFuture at 0x153d29d5abd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 173 has memoization hash be6dd4a0d16c691e32fd298812cf138a
parsl.dataflow.memoization INFO: Task 173 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/697d56fd-c7d2-4047-86cb-24674bb95b8f_isrForCrosstalkSources_871265_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/697d56fd-c7d2-4047-86cb-24674bb95b8f_isrForCrosstalkSources_871265_14.stderr
parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: <AppFuture at 0x153d29d5b690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 174 has memoization hash e5ad3c735d92324532478ebbf3fab852
parsl.dataflow.memoization INFO: Task 174 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/2b60eeba-e62c-4fb5-a1c2-b14996e0ca31_isrForCrosstalkSources_871267_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/2b60eeba-e62c-4fb5-a1c2-b14996e0ca31_isrForCrosstalkSources_871267_18.stderr
parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: <AppFuture at 0x153d29d5b9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 175 has memoization hash d7fa57c22b2928eda53378e3a9d85986
parsl.dataflow.memoization INFO: Task 175 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/949f7096-a6d1-4e9f-9dbc-601740c438c8_isrForCrosstalkSources_871266_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/949f7096-a6d1-4e9f-9dbc-601740c438c8_isrForCrosstalkSources_871266_18.stderr
parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: <AppFuture at 0x153d29d24150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 2399c0e95b1f1af8ec4028014df8c316
parsl.dataflow.memoization INFO: Task 176 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/434fd797-b9c9-4a8d-928c-fbac00a7e56e_isrForCrosstalkSources_871266_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/434fd797-b9c9-4a8d-928c-fbac00a7e56e_isrForCrosstalkSources_871266_56.stderr
parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: <AppFuture at 0x153d29d5d290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 022ba1882b69068152d4863aa7b1f071
parsl.dataflow.memoization INFO: Task 177 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/92918767-dab7-46b0-b552-f922cb1ea242_isrForCrosstalkSources_871267_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/92918767-dab7-46b0-b552-f922cb1ea242_isrForCrosstalkSources_871267_36.stderr
parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: <AppFuture at 0x153d29d5d650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 6be8b5ffcaac37372321f42ffd73bf92
parsl.dataflow.memoization INFO: Task 178 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d8b40d8f-1f85-4a95-933b-e1b3fc130508_isrForCrosstalkSources_871267_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d8b40d8f-1f85-4a95-933b-e1b3fc130508_isrForCrosstalkSources_871267_5.stderr
parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: <AppFuture at 0x153d29d5d310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 179 has memoization hash b575099d0303116ec01df6399a76c9da
parsl.dataflow.memoization INFO: Task 179 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/eafdc984-ea75-4ac9-a843-c3b30322b966_isrForCrosstalkSources_871266_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/eafdc984-ea75-4ac9-a843-c3b30322b966_isrForCrosstalkSources_871266_31.stderr
parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: <AppFuture at 0x153d29d5e350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 60bbc05daccd9054faaf35027615ea52
parsl.dataflow.memoization INFO: Task 180 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a2af5d4e-6858-4a1c-8a5e-e8f6ba50f6e2_isrForCrosstalkSources_871266_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a2af5d4e-6858-4a1c-8a5e-e8f6ba50f6e2_isrForCrosstalkSources_871266_17.stderr
parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: <AppFuture at 0x153d29d5ec90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 181 has memoization hash af7905d4e874893e43b2a15165c28c18
parsl.dataflow.memoization INFO: Task 181 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b6357928-134a-4745-8e2f-804691e17f97_isrForCrosstalkSources_871266_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b6357928-134a-4745-8e2f-804691e17f97_isrForCrosstalkSources_871266_21.stderr
parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: <AppFuture at 0x153d29d5f550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 425c1388ef9e14bbc9a66cc447f08879
parsl.dataflow.memoization INFO: Task 182 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/19e4c572-a054-489f-83f2-d0113ee0e0f4_isrForCrosstalkSources_871265_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/19e4c572-a054-489f-83f2-d0113ee0e0f4_isrForCrosstalkSources_871265_56.stderr
parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: <AppFuture at 0x153d29d5fdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 49ea08bd1d6bfa7e8d426e6d4ffc74d9
parsl.dataflow.memoization INFO: Task 183 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b326fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e5ae510c-1519-4ee1-ada9-00602996fcb1_isrForCrosstalkSources_871266_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e5ae510c-1519-4ee1-ada9-00602996fcb1_isrForCrosstalkSources_871266_11.stderr
parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: <AppFuture at 0x153d29d68a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 212217bdb91ed28d456b32c452f6cab2
parsl.dataflow.memoization INFO: Task 184 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b327060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/0fa4fced-5198-44bd-bd73-cb3d8828cc07_isrForCrosstalkSources_871265_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/0fa4fced-5198-44bd-bd73-cb3d8828cc07_isrForCrosstalkSources_871265_44.stderr
parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: <AppFuture at 0x153d29d69250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 185 has memoization hash aacf9907652d14ff3c870e654e958379
parsl.dataflow.memoization INFO: Task 185 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x153d2b3271a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/07a3e405-57a4-453f-bcf3-4b835f0c4bfc_isrForCrosstalkSources_871266_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/07a3e405-57a4-453f-bcf3-4b835f0c4bfc_isrForCrosstalkSources_871266_50.stderr
2024-03-10 19:24:48 proc_lsst.multi:146 [INFO]  found job 96597 in provider local
proc_lsst.multi INFO: found job 96597 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 186
parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism
parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out
parsl.dataflow.strategy DEBUG: Requesting 46 more blocks
parsl.executors.status_handling INFO: Scaling out by 46 blocks
parsl.executors.status_handling INFO: Allocated block ID 1
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1
2024-03-10 19:24:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
2024-03-10 19:24:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-10 19:24:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:48 proc_lsst.multi:170 [INFO]  [multi] job_id 16982456
proc_lsst.multi INFO: [multi] job_id 16982456
2024-03-10 19:24:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:48 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982456
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982456
parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16982456
parsl.executors.status_handling INFO: Allocated block ID 2
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2
2024-03-10 19:24:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
2024-03-10 19:24:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:49 proc_lsst.multi:170 [INFO]  [multi] job_id 16982457
proc_lsst.multi INFO: [multi] job_id 16982457
2024-03-10 19:24:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-10 19:24:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:49 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982457
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982457
parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16982457
parsl.executors.status_handling INFO: Allocated block ID 3
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3
2024-03-10 19:24:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
2024-03-10 19:24:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-10 19:24:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:49 proc_lsst.multi:170 [INFO]  [multi] job_id 16982458
proc_lsst.multi INFO: [multi] job_id 16982458
2024-03-10 19:24:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-10 19:24:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:49 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982458
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982458
parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16982458
parsl.executors.status_handling INFO: Allocated block ID 4
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4
2024-03-10 19:24:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
2024-03-10 19:24:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-10 19:24:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:50 proc_lsst.multi:170 [INFO]  [multi] job_id 16982459
proc_lsst.multi INFO: [multi] job_id 16982459
2024-03-10 19:24:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-10 19:24:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:50 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982459
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982459
parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16982459
parsl.executors.status_handling INFO: Allocated block ID 5
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5
2024-03-10 19:24:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
2024-03-10 19:24:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-10 19:24:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:50 proc_lsst.multi:170 [INFO]  [multi] job_id 16982461
proc_lsst.multi INFO: [multi] job_id 16982461
2024-03-10 19:24:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:24:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:50 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982461
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982461
parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16982461
parsl.executors.status_handling INFO: Allocated block ID 6
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6
2024-03-10 19:24:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
2024-03-10 19:24:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:24:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:51 proc_lsst.multi:170 [INFO]  [multi] job_id 16982462
proc_lsst.multi INFO: [multi] job_id 16982462
2024-03-10 19:24:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-10 19:24:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:51 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982462
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982462
parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16982462
parsl.executors.status_handling INFO: Allocated block ID 7
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7
2024-03-10 19:24:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
2024-03-10 19:24:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-10 19:24:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:51 proc_lsst.multi:170 [INFO]  [multi] job_id 16982463
proc_lsst.multi INFO: [multi] job_id 16982463
2024-03-10 19:24:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-10 19:24:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:51 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982463
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982463
parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16982463
parsl.executors.status_handling INFO: Allocated block ID 8
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8
2024-03-10 19:24:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
2024-03-10 19:24:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-10 19:24:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:52 proc_lsst.multi:170 [INFO]  [multi] job_id 16982464
proc_lsst.multi INFO: [multi] job_id 16982464
2024-03-10 19:24:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-10 19:24:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:52 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982464
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982464
parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16982464
parsl.executors.status_handling INFO: Allocated block ID 9
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9
2024-03-10 19:24:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
2024-03-10 19:24:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-10 19:24:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:53 proc_lsst.multi:170 [INFO]  [multi] job_id 16982465
proc_lsst.multi INFO: [multi] job_id 16982465
2024-03-10 19:24:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-10 19:24:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:53 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982465
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982465
parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16982465
parsl.executors.status_handling INFO: Allocated block ID 10
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10
2024-03-10 19:24:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
2024-03-10 19:24:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-10 19:24:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:53 proc_lsst.multi:170 [INFO]  [multi] job_id 16982466
proc_lsst.multi INFO: [multi] job_id 16982466
2024-03-10 19:24:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-10 19:24:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:53 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982466
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982466
parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16982466
parsl.executors.status_handling INFO: Allocated block ID 11
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11
2024-03-10 19:24:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
2024-03-10 19:24:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-10 19:24:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:53 proc_lsst.multi:170 [INFO]  [multi] job_id 16982467
proc_lsst.multi INFO: [multi] job_id 16982467
2024-03-10 19:24:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-10 19:24:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:53 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982467
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982467
parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16982467
parsl.executors.status_handling INFO: Allocated block ID 12
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12
2024-03-10 19:24:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
2024-03-10 19:24:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-10 19:24:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:54 proc_lsst.multi:170 [INFO]  [multi] job_id 16982468
proc_lsst.multi INFO: [multi] job_id 16982468
2024-03-10 19:24:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-10 19:24:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:54 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982468
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982468
parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16982468
parsl.executors.status_handling INFO: Allocated block ID 13
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13
2024-03-10 19:24:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
2024-03-10 19:24:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-10 19:24:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:54 proc_lsst.multi:170 [INFO]  [multi] job_id 16982469
proc_lsst.multi INFO: [multi] job_id 16982469
2024-03-10 19:24:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-10 19:24:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:54 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982469
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982469
parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16982469
parsl.executors.status_handling INFO: Allocated block ID 14
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14
2024-03-10 19:24:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
2024-03-10 19:24:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-10 19:24:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:55 proc_lsst.multi:170 [INFO]  [multi] job_id 16982470
proc_lsst.multi INFO: [multi] job_id 16982470
2024-03-10 19:24:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-10 19:24:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:55 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982470
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982470
parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16982470
parsl.executors.status_handling INFO: Allocated block ID 15
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15
2024-03-10 19:24:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
2024-03-10 19:24:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-10 19:24:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:56 proc_lsst.multi:170 [INFO]  [multi] job_id 16982471
proc_lsst.multi INFO: [multi] job_id 16982471
2024-03-10 19:24:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-10 19:24:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:56 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982471
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982471
parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16982471
parsl.executors.status_handling INFO: Allocated block ID 16
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16
2024-03-10 19:24:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
2024-03-10 19:24:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-10 19:24:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:56 proc_lsst.multi:170 [INFO]  [multi] job_id 16982472
proc_lsst.multi INFO: [multi] job_id 16982472
2024-03-10 19:24:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-10 19:24:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:56 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982472
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982472
parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16982472
parsl.executors.status_handling INFO: Allocated block ID 17
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17
2024-03-10 19:24:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
2024-03-10 19:24:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-10 19:24:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:57 proc_lsst.multi:170 [INFO]  [multi] job_id 16982473
proc_lsst.multi INFO: [multi] job_id 16982473
2024-03-10 19:24:57 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-10 19:24:57 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:57 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982473
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982473
parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16982473
parsl.executors.status_handling INFO: Allocated block ID 18
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18
2024-03-10 19:24:57 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
2024-03-10 19:24:57 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:57 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:57 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-10 19:24:57 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:57 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:58 proc_lsst.multi:170 [INFO]  [multi] job_id 16982474
proc_lsst.multi INFO: [multi] job_id 16982474
2024-03-10 19:24:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-10 19:24:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:58 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982474
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982474
parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16982474
parsl.executors.status_handling INFO: Allocated block ID 19
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19
2024-03-10 19:24:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
2024-03-10 19:24:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-10 19:24:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:58 proc_lsst.multi:170 [INFO]  [multi] job_id 16982475
proc_lsst.multi INFO: [multi] job_id 16982475
2024-03-10 19:24:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-10 19:24:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:58 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982475
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982475
parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16982475
parsl.executors.status_handling INFO: Allocated block ID 20
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20
2024-03-10 19:24:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
2024-03-10 19:24:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-10 19:24:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:58 proc_lsst.multi:170 [INFO]  [multi] job_id 16982476
proc_lsst.multi INFO: [multi] job_id 16982476
2024-03-10 19:24:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-10 19:24:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:58 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982476
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982476
parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16982476
parsl.executors.status_handling INFO: Allocated block ID 21
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21
2024-03-10 19:24:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
2024-03-10 19:24:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-10 19:24:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:58 proc_lsst.multi:170 [INFO]  [multi] job_id 16982477
proc_lsst.multi INFO: [multi] job_id 16982477
2024-03-10 19:24:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-10 19:24:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:58 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982477
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982477
parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16982477
parsl.executors.status_handling INFO: Allocated block ID 22
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22
2024-03-10 19:24:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
2024-03-10 19:24:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-10 19:24:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:59 proc_lsst.multi:170 [INFO]  [multi] job_id 16982478
proc_lsst.multi INFO: [multi] job_id 16982478
2024-03-10 19:24:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-10 19:24:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:59 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982478
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982478
parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16982478
parsl.executors.status_handling INFO: Allocated block ID 23
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23
2024-03-10 19:24:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
2024-03-10 19:24:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-10 19:24:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:24:59 proc_lsst.multi:170 [INFO]  [multi] job_id 16982479
proc_lsst.multi INFO: [multi] job_id 16982479
2024-03-10 19:24:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-10 19:24:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:59 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982479
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982479
parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16982479
parsl.executors.status_handling INFO: Allocated block ID 24
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24
2024-03-10 19:24:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
2024-03-10 19:24:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:24:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:24:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:24:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:24:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-10 19:24:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:24:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:00 proc_lsst.multi:170 [INFO]  [multi] job_id 16982480
proc_lsst.multi INFO: [multi] job_id 16982480
2024-03-10 19:25:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-10 19:25:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982480
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982480
parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16982480
parsl.executors.status_handling INFO: Allocated block ID 25
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25
2024-03-10 19:25:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:00 proc_lsst.multi:170 [INFO]  [multi] job_id 16982481
proc_lsst.multi INFO: [multi] job_id 16982481
2024-03-10 19:25:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-10 19:25:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982481
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982481
parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16982481
parsl.executors.status_handling INFO: Allocated block ID 26
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26
2024-03-10 19:25:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:00 proc_lsst.multi:170 [INFO]  [multi] job_id 16982482
proc_lsst.multi INFO: [multi] job_id 16982482
2024-03-10 19:25:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-10 19:25:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982482
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982482
parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16982482
parsl.executors.status_handling INFO: Allocated block ID 27
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27
2024-03-10 19:25:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:00 proc_lsst.multi:170 [INFO]  [multi] job_id 16982483
proc_lsst.multi INFO: [multi] job_id 16982483
2024-03-10 19:25:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-10 19:25:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982483
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982483
parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16982483
parsl.executors.status_handling INFO: Allocated block ID 28
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28
2024-03-10 19:25:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:00 proc_lsst.multi:170 [INFO]  [multi] job_id 16982484
proc_lsst.multi INFO: [multi] job_id 16982484
2024-03-10 19:25:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-10 19:25:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982484
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982484
parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16982484
parsl.executors.status_handling INFO: Allocated block ID 29
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29
2024-03-10 19:25:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-10 19:25:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:01 proc_lsst.multi:170 [INFO]  [multi] job_id 16982485
proc_lsst.multi INFO: [multi] job_id 16982485
2024-03-10 19:25:01 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-10 19:25:01 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:01 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982485
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982485
parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16982485
parsl.executors.status_handling INFO: Allocated block ID 30
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30
2024-03-10 19:25:01 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
2024-03-10 19:25:01 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:01 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-10 19:25:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:01 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20802451ad18f55d11ca03a36a25bc0e with result from task 1
2024-03-10 19:25:01 proc_lsst.multi:170 [INFO]  [multi] job_id 16982486
proc_lsst.multi INFO: [multi] job_id 16982486
2024-03-10 19:25:01 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:01 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:01 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 16982486
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16982486
parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16982486
parsl.executors.status_handling INFO: Allocated block ID 31
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31
2024-03-10 19:25:01 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
2024-03-10 19:25:01 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:01 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:01 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-10 19:25:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:01 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:01 proc_lsst.multi:170 [INFO]  [multi] job_id 16982487
proc_lsst.multi INFO: [multi] job_id 16982487
2024-03-10 19:25:01 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:01 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:01 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16982487
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16982487
parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16982487
parsl.executors.status_handling INFO: Allocated block ID 32
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32
2024-03-10 19:25:01 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
2024-03-10 19:25:01 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:01 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:01 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:01 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow DEBUG: Task 1 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e3ad8990-1aae-4162-9f21-f2fd0cde30ba_isrForCrosstalkSources_871267_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e3ad8990-1aae-4162-9f21-f2fd0cde30ba_isrForCrosstalkSources_871267_33.stderr
parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 622a7f851754d3d31dadc87b23caa535 with result from task 0
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow DEBUG: Task 0 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/13d2c775-ec0e-40ea-be1c-f93f71d524ab_isrForCrosstalkSources_871265_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/13d2c775-ec0e-40ea-be1c-f93f71d524ab_isrForCrosstalkSources_871265_49.stderr
2024-03-10 19:25:02 proc_lsst.multi:170 [INFO]  [multi] job_id 16982488
proc_lsst.multi INFO: [multi] job_id 16982488
2024-03-10 19:25:02 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-10 19:25:02 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:02 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16982488
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16982488
parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16982488
parsl.executors.status_handling INFO: Allocated block ID 33
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33
2024-03-10 19:25:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
2024-03-10 19:25:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:02 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-10 19:25:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:03 proc_lsst.multi:170 [INFO]  [multi] job_id 16982489
proc_lsst.multi INFO: [multi] job_id 16982489
2024-03-10 19:25:03 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-10 19:25:03 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:03 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16982489
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16982489
parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16982489
parsl.executors.status_handling INFO: Allocated block ID 34
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34
2024-03-10 19:25:03 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
2024-03-10 19:25:03 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:03 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:03 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-10 19:25:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:03 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:03 proc_lsst.multi:170 [INFO]  [multi] job_id 16982490
proc_lsst.multi INFO: [multi] job_id 16982490
2024-03-10 19:25:03 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-10 19:25:03 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:03 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16982490
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16982490
parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16982490
parsl.executors.status_handling INFO: Allocated block ID 35
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35
2024-03-10 19:25:03 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
2024-03-10 19:25:03 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:03 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:03 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-10 19:25:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:03 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:04 proc_lsst.multi:170 [INFO]  [multi] job_id 16982491
proc_lsst.multi INFO: [multi] job_id 16982491
2024-03-10 19:25:04 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:04 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:04 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 16982491
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16982491
parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16982491
parsl.executors.status_handling INFO: Allocated block ID 36
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36
2024-03-10 19:25:04 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
2024-03-10 19:25:04 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:04 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:04 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:04 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:04 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-10 19:25:04 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:04 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:06 proc_lsst.multi:170 [INFO]  [multi] job_id 16982492
proc_lsst.multi INFO: [multi] job_id 16982492
2024-03-10 19:25:06 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:06 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:06 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982492
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982492
parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16982492
parsl.executors.status_handling INFO: Allocated block ID 37
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37
2024-03-10 19:25:06 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
2024-03-10 19:25:06 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:06 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:06 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:06 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:06 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:06 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:06 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:07 proc_lsst.multi:170 [INFO]  [multi] job_id 16982493
proc_lsst.multi INFO: [multi] job_id 16982493
2024-03-10 19:25:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-10 19:25:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:07 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982493
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982493
parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16982493
parsl.executors.status_handling INFO: Allocated block ID 38
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38
2024-03-10 19:25:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
2024-03-10 19:25:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:07 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-10 19:25:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:07 proc_lsst.multi:170 [INFO]  [multi] job_id 16982494
proc_lsst.multi INFO: [multi] job_id 16982494
2024-03-10 19:25:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-10 19:25:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:07 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982494
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982494
parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16982494
parsl.executors.status_handling INFO: Allocated block ID 39
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39
2024-03-10 19:25:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
2024-03-10 19:25:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:07 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-10 19:25:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:08 proc_lsst.multi:170 [INFO]  [multi] job_id 16982495
proc_lsst.multi INFO: [multi] job_id 16982495
2024-03-10 19:25:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-10 19:25:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:08 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982495
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982495
parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16982495
parsl.executors.status_handling INFO: Allocated block ID 40
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40
2024-03-10 19:25:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:08 proc_lsst.multi:170 [INFO]  [multi] job_id 16982496
proc_lsst.multi INFO: [multi] job_id 16982496
2024-03-10 19:25:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:08 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982496
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982496
parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16982496
parsl.executors.status_handling INFO: Allocated block ID 41
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41
2024-03-10 19:25:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:08 proc_lsst.multi:170 [INFO]  [multi] job_id 16982497
proc_lsst.multi INFO: [multi] job_id 16982497
2024-03-10 19:25:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-10 19:25:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:08 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982497
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982497
parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16982497
parsl.executors.status_handling INFO: Allocated block ID 42
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42
2024-03-10 19:25:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:08 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-10 19:25:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:09 proc_lsst.multi:170 [INFO]  [multi] job_id 16982498
proc_lsst.multi INFO: [multi] job_id 16982498
2024-03-10 19:25:09 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-10 19:25:09 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:09 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982498
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982498
parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16982498
parsl.executors.status_handling INFO: Allocated block ID 43
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43
2024-03-10 19:25:09 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
2024-03-10 19:25:09 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:09 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:09 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:09 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-10 19:25:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:09 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:10 proc_lsst.multi:170 [INFO]  [multi] job_id 16982499
proc_lsst.multi INFO: [multi] job_id 16982499
2024-03-10 19:25:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-10 19:25:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:10 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982499
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982499
parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16982499
parsl.executors.status_handling INFO: Allocated block ID 44
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44
2024-03-10 19:25:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
2024-03-10 19:25:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:10 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:10 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-10 19:25:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:11 proc_lsst.multi:170 [INFO]  [multi] job_id 16982500
proc_lsst.multi INFO: [multi] job_id 16982500
2024-03-10 19:25:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-10 19:25:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:11 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982500
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982500
parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16982500
parsl.executors.status_handling INFO: Allocated block ID 45
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45
2024-03-10 19:25:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
2024-03-10 19:25:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:11 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:11 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-10 19:25:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:11 proc_lsst.multi:170 [INFO]  [multi] job_id 16982501
proc_lsst.multi INFO: [multi] job_id 16982501
2024-03-10 19:25:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-10 19:25:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:11 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982501
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982501
parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16982501
parsl.executors.status_handling INFO: Allocated block ID 46
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46
2024-03-10 19:25:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
2024-03-10 19:25:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-10 19:25:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-10 19:25:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-10 19:25:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-10 19:25:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-10 19:25:11 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-10 19:25:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-10 19:25:11 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-10 19:25:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-10 19:25:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.10,198.48.92.26,169.254.95.120,n3010,10.64.129.10,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54911 --result_port=54720 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-10 19:25:12 proc_lsst.multi:170 [INFO]  [multi] job_id 16982502
proc_lsst.multi INFO: [multi] job_id 16982502
2024-03-10 19:25:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-10 19:25:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-10 19:25:12 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 16982502
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16982502
parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16982502
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 184
parsl.dataflow.strategy DEBUG: Executor multi has 184 active tasks, 1/46 running/pending blocks, and 26 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5e9c17f39e37a1a7f9a0d6ace0c7dcbc with result from task 3
parsl.dataflow.dflow DEBUG: Task 3 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fbbcbd9e-646c-4954-8a11-72efa5d4cd2d_isrForCrosstalkSources_871267_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fbbcbd9e-646c-4954-8a11-72efa5d4cd2d_isrForCrosstalkSources_871267_4.stderr
parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5666406ad03074ad8d6b2a1cac783fd4 with result from task 2
parsl.dataflow.dflow DEBUG: Task 2 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0a48706a-caa3-47e7-82cd-be996e44fcc9_isrForCrosstalkSources_871267_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0a48706a-caa3-47e7-82cd-be996e44fcc9_isrForCrosstalkSources_871267_47.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 182
parsl.dataflow.strategy DEBUG: Executor multi has 182 active tasks, 1/46 running/pending blocks, and 26 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
2024-03-10 19:25:22 proc_lsst.multi:146 [INFO]  found job 96597 in provider local
proc_lsst.multi INFO: found job 96597 in provider local
2024-03-10 19:25:22 proc_lsst.multi:146 [INFO]  found job 16982456 in provider astro
proc_lsst.multi INFO: found job 16982456 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:23 proc_lsst.multi:146 [INFO]  found job 16982457 in provider astro
proc_lsst.multi INFO: found job 16982457 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:23 proc_lsst.multi:146 [INFO]  found job 16982458 in provider astro
proc_lsst.multi INFO: found job 16982458 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:23 proc_lsst.multi:146 [INFO]  found job 16982459 in provider astro
proc_lsst.multi INFO: found job 16982459 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:24 proc_lsst.multi:146 [INFO]  found job 16982461 in provider astro
proc_lsst.multi INFO: found job 16982461 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:24 proc_lsst.multi:146 [INFO]  found job 16982462 in provider astro
proc_lsst.multi INFO: found job 16982462 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:24 proc_lsst.multi:146 [INFO]  found job 16982463 in provider astro
proc_lsst.multi INFO: found job 16982463 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:25 proc_lsst.multi:146 [INFO]  found job 16982464 in provider astro
proc_lsst.multi INFO: found job 16982464 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:25 proc_lsst.multi:146 [INFO]  found job 16982465 in provider astro
proc_lsst.multi INFO: found job 16982465 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:25 proc_lsst.multi:146 [INFO]  found job 16982466 in provider astro
proc_lsst.multi INFO: found job 16982466 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:25 proc_lsst.multi:146 [INFO]  found job 16982467 in provider astro
proc_lsst.multi INFO: found job 16982467 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:26 proc_lsst.multi:146 [INFO]  found job 16982468 in provider astro
proc_lsst.multi INFO: found job 16982468 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:26 proc_lsst.multi:146 [INFO]  found job 16982469 in provider astro
proc_lsst.multi INFO: found job 16982469 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:26 proc_lsst.multi:146 [INFO]  found job 16982470 in provider astro
proc_lsst.multi INFO: found job 16982470 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:27 proc_lsst.multi:146 [INFO]  found job 16982471 in provider astro
proc_lsst.multi INFO: found job 16982471 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:27 proc_lsst.multi:146 [INFO]  found job 16982472 in provider astro
proc_lsst.multi INFO: found job 16982472 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:27 proc_lsst.multi:146 [INFO]  found job 16982473 in provider astro
proc_lsst.multi INFO: found job 16982473 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:27 proc_lsst.multi:146 [INFO]  found job 16982474 in provider astro
proc_lsst.multi INFO: found job 16982474 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:28 proc_lsst.multi:146 [INFO]  found job 16982475 in provider astro
proc_lsst.multi INFO: found job 16982475 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 71cc386eb3d030f683d5a853e21334ec with result from task 29
parsl.dataflow.dflow DEBUG: Task 29 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3c63998c-c5c7-4de1-aad3-049914f4bd32_isrForCrosstalkSources_871265_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3c63998c-c5c7-4de1-aad3-049914f4bd32_isrForCrosstalkSources_871265_15.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:28 proc_lsst.multi:146 [INFO]  found job 16982476 in provider astro
proc_lsst.multi INFO: found job 16982476 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ff7d2f7116d9308c0e35deb2f7b3061 with result from task 28
parsl.dataflow.dflow DEBUG: Task 28 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/e85a2a5c-09ca-4ae8-987d-60b0d5cd14d0_isrForCrosstalkSources_871265_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/e85a2a5c-09ca-4ae8-987d-60b0d5cd14d0_isrForCrosstalkSources_871265_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:28 proc_lsst.multi:146 [INFO]  found job 16982477 in provider astro
proc_lsst.multi INFO: found job 16982477 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:28 proc_lsst.multi:146 [INFO]  found job 16982478 in provider astro
proc_lsst.multi INFO: found job 16982478 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:29 proc_lsst.multi:146 [INFO]  found job 16982479 in provider astro
proc_lsst.multi INFO: found job 16982479 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:29 proc_lsst.multi:146 [INFO]  found job 16982480 in provider astro
proc_lsst.multi INFO: found job 16982480 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:29 proc_lsst.multi:146 [INFO]  found job 16982481 in provider astro
proc_lsst.multi INFO: found job 16982481 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:29 proc_lsst.multi:146 [INFO]  found job 16982482 in provider astro
proc_lsst.multi INFO: found job 16982482 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:30 proc_lsst.multi:146 [INFO]  found job 16982483 in provider astro
proc_lsst.multi INFO: found job 16982483 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:30 proc_lsst.multi:146 [INFO]  found job 16982484 in provider astro
proc_lsst.multi INFO: found job 16982484 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:31 proc_lsst.multi:146 [INFO]  found job 16982485 in provider astro
proc_lsst.multi INFO: found job 16982485 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:31 proc_lsst.multi:146 [INFO]  found job 16982486 in provider astro
proc_lsst.multi INFO: found job 16982486 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:32 proc_lsst.multi:146 [INFO]  found job 16982487 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982487 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:32 proc_lsst.multi:146 [INFO]  found job 16982488 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982488 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:32 proc_lsst.multi:146 [INFO]  found job 16982489 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982489 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:33 proc_lsst.multi:146 [INFO]  found job 16982490 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982490 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:33 proc_lsst.multi:146 [INFO]  found job 16982491 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982491 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:33 proc_lsst.multi:146 [INFO]  found job 16982492 in provider escience_a40
proc_lsst.multi INFO: found job 16982492 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:34 proc_lsst.multi:146 [INFO]  found job 16982493 in provider escience_a40
proc_lsst.multi INFO: found job 16982493 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:34 proc_lsst.multi:146 [INFO]  found job 16982494 in provider escience_a40
proc_lsst.multi INFO: found job 16982494 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:35 proc_lsst.multi:146 [INFO]  found job 16982495 in provider escience_a40
proc_lsst.multi INFO: found job 16982495 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:35 proc_lsst.multi:146 [INFO]  found job 16982496 in provider escience_a40
proc_lsst.multi INFO: found job 16982496 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:35 proc_lsst.multi:146 [INFO]  found job 16982497 in provider escience_a40
proc_lsst.multi INFO: found job 16982497 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:36 proc_lsst.multi:146 [INFO]  found job 16982498 in provider escience_a40
proc_lsst.multi INFO: found job 16982498 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:38 proc_lsst.multi:146 [INFO]  found job 16982499 in provider escience_a40
proc_lsst.multi INFO: found job 16982499 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:38 proc_lsst.multi:146 [INFO]  found job 16982500 in provider escience_a40
proc_lsst.multi INFO: found job 16982500 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:38 proc_lsst.multi:146 [INFO]  found job 16982501 in provider escience_a40
proc_lsst.multi INFO: found job 16982501 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:39 proc_lsst.multi:146 [INFO]  found job 16982502 in provider escience_a40
proc_lsst.multi INFO: found job 16982502 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dba3f03c5a9c6f7a526617187f9d40f1 with result from task 6
parsl.dataflow.dflow DEBUG: Task 6 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b8c90a3c-2439-475a-89ad-d0621b8ae117_isrForCrosstalkSources_871267_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b8c90a3c-2439-475a-89ad-d0621b8ae117_isrForCrosstalkSources_871267_39.stderr
parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6ce277dae347b0a32c614c7bf94efd38 with result from task 18
parsl.dataflow.dflow DEBUG: Task 18 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/659599ca-b49c-486c-be8f-6b2b1c8de4b4_isrForCrosstalkSources_871265_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/659599ca-b49c-486c-be8f-6b2b1c8de4b4_isrForCrosstalkSources_871265_45.stderr
parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47a17750b1bd769c146b992d9d811235 with result from task 17
parsl.dataflow.dflow DEBUG: Task 17 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b2b7cabe-0d5e-4420-9b32-74e0d4b99f02_isrForCrosstalkSources_871265_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b2b7cabe-0d5e-4420-9b32-74e0d4b99f02_isrForCrosstalkSources_871265_23.stderr
parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c4c1b0bd2c48d1e86c20fcdc805af577 with result from task 27
parsl.dataflow.dflow DEBUG: Task 27 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c78ce7ec-9f64-423a-b920-d8b0cc456a42_isrForCrosstalkSources_871265_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c78ce7ec-9f64-423a-b920-d8b0cc456a42_isrForCrosstalkSources_871265_4.stderr
parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 81877b80a91a5249f966dfddbc2ddf64 with result from task 11
parsl.dataflow.dflow DEBUG: Task 11 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/13db6717-de90-4eac-991c-e804725090bb_isrForCrosstalkSources_871267_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/13db6717-de90-4eac-991c-e804725090bb_isrForCrosstalkSources_871267_59.stderr
parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c665ced78181a0161521fb006f8d99a3 with result from task 25
parsl.dataflow.dflow DEBUG: Task 25 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/3fc7e56b-072d-4635-9139-5d61a19b6e86_isrForCrosstalkSources_871267_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/3fc7e56b-072d-4635-9139-5d61a19b6e86_isrForCrosstalkSources_871267_11.stderr
parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7fabd2050023c3e0346511df492965aa with result from task 20
parsl.dataflow.dflow DEBUG: Task 20 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d8b3def0-8bcf-42f2-a155-4bf92860bc6e_isrForCrosstalkSources_871267_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d8b3def0-8bcf-42f2-a155-4bf92860bc6e_isrForCrosstalkSources_871267_26.stderr
parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 349e2e2963cdff8af49096b96200d178 with result from task 12
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.dflow DEBUG: Task 12 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/faba3ec8-cf43-4b0d-a560-74009baedb7e_isrForCrosstalkSources_871267_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/faba3ec8-cf43-4b0d-a560-74009baedb7e_isrForCrosstalkSources_871267_23.stderr
parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eeb64e015b80989f66df0837b1a30162 with result from task 13
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 168
parsl.dataflow.dflow DEBUG: Task 13 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b73a0d2b-c48e-4b39-852a-f774a61fbc32_isrForCrosstalkSources_871267_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b73a0d2b-c48e-4b39-852a-f774a61fbc32_isrForCrosstalkSources_871267_55.stderr
parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 86ca76c247780df5a14dd5c5a6b82274 with result from task 14
parsl.dataflow.strategy DEBUG: Executor multi has 168 active tasks, 47/0 running/pending blocks, and 46 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 168
parsl.dataflow.dflow DEBUG: Task 14 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/44ab5d5c-8386-4229-b685-1c8467dcacce_isrForCrosstalkSources_871267_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/44ab5d5c-8386-4229-b685-1c8467dcacce_isrForCrosstalkSources_871267_28.stderr
parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5914bd0f7b8f91bc9ad485510977eb05 with result from task 21
parsl.dataflow.strategy DEBUG: Executor multi has 168 active tasks, 47/0 running/pending blocks, and 46 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.dflow DEBUG: Task 21 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/cef70236-a502-4e38-bf44-b2a326743040_isrForCrosstalkSources_871267_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/cef70236-a502-4e38-bf44-b2a326743040_isrForCrosstalkSources_871267_27.stderr
parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1fd4aaeb2f945cf30ba3f4fc0e1b568e with result from task 22
parsl.dataflow.dflow DEBUG: Task 22 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4ceb96ca-36c0-490d-ae8a-5c875f3809be_isrForCrosstalkSources_871266_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4ceb96ca-36c0-490d-ae8a-5c875f3809be_isrForCrosstalkSources_871266_25.stderr
parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5cbf7d4b5b1bdf075d42e12ec8d5640c with result from task 4
parsl.dataflow.dflow DEBUG: Task 4 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/03d07d67-be23-4de3-a5c9-8c2d492f38f6_isrForCrosstalkSources_871265_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/03d07d67-be23-4de3-a5c9-8c2d492f38f6_isrForCrosstalkSources_871265_59.stderr
parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 14ecff33bb36d01bf6d1f129b7ec0c41 with result from task 9
parsl.dataflow.dflow DEBUG: Task 9 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/c3fb05c4-3faa-4932-88f6-63b9b997865f_isrForCrosstalkSources_871266_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/c3fb05c4-3faa-4932-88f6-63b9b997865f_isrForCrosstalkSources_871266_1.stderr
parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd7ac59c6d512468c7b0881ac20498df with result from task 23
parsl.dataflow.dflow DEBUG: Task 23 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a4d77a1e-83a6-4543-9058-27bdb014abfa_isrForCrosstalkSources_871266_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a4d77a1e-83a6-4543-9058-27bdb014abfa_isrForCrosstalkSources_871266_29.stderr
parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cae0c782add39d3da6ed7047d8be3445 with result from task 24
parsl.dataflow.dflow DEBUG: Task 24 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a9852d3a-299f-4d1c-a38b-e64023853732_isrForCrosstalkSources_871266_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a9852d3a-299f-4d1c-a38b-e64023853732_isrForCrosstalkSources_871266_54.stderr
parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e4f153eac18e87c36d60e5af6417e4b9 with result from task 16
parsl.dataflow.dflow DEBUG: Task 16 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b644ffa9-c400-499c-a86d-7319e6eaab21_isrForCrosstalkSources_871266_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b644ffa9-c400-499c-a86d-7319e6eaab21_isrForCrosstalkSources_871266_16.stderr
parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry be058a1ebce44bb312170f2017a57231 with result from task 10
parsl.dataflow.dflow DEBUG: Task 10 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e83c2e36-fb6c-42fe-a0cc-3d8ea05217f6_isrForCrosstalkSources_871266_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e83c2e36-fb6c-42fe-a0cc-3d8ea05217f6_isrForCrosstalkSources_871266_62.stderr
parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0cbd4b89b5d4fd575578420b1e9fbfeb with result from task 15
parsl.dataflow.dflow DEBUG: Task 15 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5eefb463-97ba-4476-86de-3329a2dffc63_isrForCrosstalkSources_871266_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5eefb463-97ba-4476-86de-3329a2dffc63_isrForCrosstalkSources_871266_58.stderr
parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5360b5562c9970d4ec9103a6ffd66b51 with result from task 8
parsl.dataflow.dflow DEBUG: Task 8 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4c7491e8-dfa4-457c-bcfa-c16a9df0ca26_isrForCrosstalkSources_871266_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4c7491e8-dfa4-457c-bcfa-c16a9df0ca26_isrForCrosstalkSources_871266_42.stderr
parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7bdca0cf95555fdcfd035228b098278 with result from task 19
parsl.dataflow.dflow DEBUG: Task 19 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/7d41a93a-b0b5-48b0-92cc-2f98c606b294_isrForCrosstalkSources_871266_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/7d41a93a-b0b5-48b0-92cc-2f98c606b294_isrForCrosstalkSources_871266_57.stderr
parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5035ba8f17b9ed86ae4e6f40835691b with result from task 50
parsl.dataflow.dflow DEBUG: Task 50 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b4f3c12b-94c3-4abb-9e32-c90c15b9b69b_isrForCrosstalkSources_871267_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b4f3c12b-94c3-4abb-9e32-c90c15b9b69b_isrForCrosstalkSources_871267_20.stderr
parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 123cffbbc0ad760f515c1e78949e7925 with result from task 51
parsl.dataflow.dflow DEBUG: Task 51 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d74b10ac-5296-4207-9889-5e45d8dcce33_isrForCrosstalkSources_871267_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d74b10ac-5296-4207-9889-5e45d8dcce33_isrForCrosstalkSources_871267_34.stderr
parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab9a8454445ff502a6b1cb39dfef223f with result from task 26
parsl.dataflow.dflow DEBUG: Task 26 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/97378aeb-8266-46dc-bea0-4cb29c5c8d38_isrForCrosstalkSources_871267_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/97378aeb-8266-46dc-bea0-4cb29c5c8d38_isrForCrosstalkSources_871267_24.stderr
parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 87e3a02195b6721180d5177611e24407 with result from task 7
parsl.dataflow.dflow DEBUG: Task 7 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b85b4baa-d9b4-47b0-a6cc-0d0e9ca12a68_isrForCrosstalkSources_871267_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b85b4baa-d9b4-47b0-a6cc-0d0e9ca12a68_isrForCrosstalkSources_871267_57.stderr
parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e209a5feb60b804a9b716d61e340b379 with result from task 5
parsl.dataflow.dflow DEBUG: Task 5 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/47221143-6556-4824-8759-69baad029ede_isrForCrosstalkSources_871266_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/47221143-6556-4824-8759-69baad029ede_isrForCrosstalkSources_871266_44.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 154
parsl.dataflow.strategy DEBUG: Executor multi has 154 active tasks, 47/0 running/pending blocks, and 46 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 327523c011be0e97bf4ad69389b428f7 with result from task 41
parsl.dataflow.dflow DEBUG: Task 41 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8dac5e52-ad75-44a9-9548-b6f0a75ca510_isrForCrosstalkSources_871265_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8dac5e52-ad75-44a9-9548-b6f0a75ca510_isrForCrosstalkSources_871265_22.stderr
parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d0f305278d45dae49ff3fba9bb3532bc with result from task 30
parsl.dataflow.dflow DEBUG: Task 30 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e9149a42-e609-4bb6-86b8-fb2064946e62_isrForCrosstalkSources_871266_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e9149a42-e609-4bb6-86b8-fb2064946e62_isrForCrosstalkSources_871266_10.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 152
parsl.dataflow.strategy DEBUG: Executor multi has 152 active tasks, 47/0 running/pending blocks, and 46 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 976bba566082ef6e85ebd8ea59e18f99 with result from task 35
parsl.dataflow.dflow DEBUG: Task 35 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/6fc2c924-b88d-4fc9-8027-8462c8bf8bf8_isrForCrosstalkSources_871267_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/6fc2c924-b88d-4fc9-8027-8462c8bf8bf8_isrForCrosstalkSources_871267_62.stderr
parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a3a2d9d221ec5954320da89da95e5500 with result from task 34
parsl.dataflow.dflow DEBUG: Task 34 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/daaad190-3862-48f3-88ce-aa1025df228b_isrForCrosstalkSources_871267_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/daaad190-3862-48f3-88ce-aa1025df228b_isrForCrosstalkSources_871267_17.stderr
parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6e04a6932acdb9e158965ee8f2da6f8d with result from task 36
parsl.dataflow.dflow DEBUG: Task 36 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/de632ea0-cee2-403b-b865-61bae1123a50_isrForCrosstalkSources_871266_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/de632ea0-cee2-403b-b865-61bae1123a50_isrForCrosstalkSources_871266_36.stderr
parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1fc113d3bafe0ebec05c124432b5c81f with result from task 38
parsl.dataflow.dflow DEBUG: Task 38 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/974305a6-7d74-49b6-b8ef-9711123ce494_isrForCrosstalkSources_871265_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/974305a6-7d74-49b6-b8ef-9711123ce494_isrForCrosstalkSources_871265_35.stderr
parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 31077c751aad9300a18a8e0260d485b8 with result from task 39
parsl.dataflow.dflow DEBUG: Task 39 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/54ae2448-a97d-4df2-9ae7-483c2b7d90c6_isrForCrosstalkSources_871266_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/54ae2448-a97d-4df2-9ae7-483c2b7d90c6_isrForCrosstalkSources_871266_23.stderr
parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5d3b1e5f16f13a87d09bce653727ef2 with result from task 33
parsl.dataflow.dflow DEBUG: Task 33 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4d835c42-7f83-4ff0-8f73-0ff0c8251ec4_isrForCrosstalkSources_871265_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4d835c42-7f83-4ff0-8f73-0ff0c8251ec4_isrForCrosstalkSources_871265_16.stderr
parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eec5814afbf7635568eb11825d5749f4 with result from task 40
parsl.dataflow.dflow DEBUG: Task 40 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/42a075bd-db97-42ce-ad23-5bddb897bd72_isrForCrosstalkSources_871265_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/42a075bd-db97-42ce-ad23-5bddb897bd72_isrForCrosstalkSources_871265_42.stderr
parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32a579fe8afb95e0b652a1da19ce0383 with result from task 32
parsl.dataflow.dflow DEBUG: Task 32 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/44a219bc-0e48-4af1-93d3-5c75d1beccbb_isrForCrosstalkSources_871265_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/44a219bc-0e48-4af1-93d3-5c75d1beccbb_isrForCrosstalkSources_871265_46.stderr
parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02eb55797463267df4f5191d40cc4d5b with result from task 31
parsl.dataflow.dflow DEBUG: Task 31 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/9cdb5abc-10e9-4d90-9b2e-402c82c5d5a8_isrForCrosstalkSources_871265_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/9cdb5abc-10e9-4d90-9b2e-402c82c5d5a8_isrForCrosstalkSources_871265_18.stderr
parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0cdc79a99d4618cf619017fab8621bc9 with result from task 37
parsl.dataflow.dflow DEBUG: Task 37 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/aa3f36bd-2c84-43a1-a3b5-ed4e3e2e2336_isrForCrosstalkSources_871265_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/aa3f36bd-2c84-43a1-a3b5-ed4e3e2e2336_isrForCrosstalkSources_871265_55.stderr
parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f6930d8b095476bf7cd3d2fc5c4c766 with result from task 60
parsl.dataflow.dflow DEBUG: Task 60 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/1e30023a-5e51-4bf9-809e-50d17b9e02e1_isrForCrosstalkSources_871266_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/1e30023a-5e51-4bf9-809e-50d17b9e02e1_isrForCrosstalkSources_871266_5.stderr
parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dab541f64d142326874f4a2eee66acb9 with result from task 68
parsl.dataflow.dflow DEBUG: Task 68 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/c6f0ca73-a56a-488e-82e8-90191a8d5a22_isrForCrosstalkSources_871266_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/c6f0ca73-a56a-488e-82e8-90191a8d5a22_isrForCrosstalkSources_871266_45.stderr
parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8237d50639392fe5afaeb57d21c2d9e with result from task 69
parsl.dataflow.dflow DEBUG: Task 69 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4fb10d2c-430c-4cdb-91dc-f2c121ef85da_isrForCrosstalkSources_871266_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/4fb10d2c-430c-4cdb-91dc-f2c121ef85da_isrForCrosstalkSources_871266_55.stderr
parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e76a9b3ae780d09ece5faf72a7d0a6d5 with result from task 61
parsl.dataflow.dflow DEBUG: Task 61 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/abba4e0e-fb71-4584-8707-366d42dece6a_isrForCrosstalkSources_871266_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/abba4e0e-fb71-4584-8707-366d42dece6a_isrForCrosstalkSources_871266_46.stderr
parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33d39b3ac2407edc6b248ec53dfba7b4 with result from task 52
parsl.dataflow.dflow DEBUG: Task 52 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/23803679-ece2-482f-a196-942f5fe46322_isrForCrosstalkSources_871266_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/23803679-ece2-482f-a196-942f5fe46322_isrForCrosstalkSources_871266_39.stderr
parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b59f81d999758cfa31802e88cf1797ce with result from task 70
parsl.dataflow.dflow DEBUG: Task 70 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b5f5da4f-4e44-44db-836e-d414b822c9c9_isrForCrosstalkSources_871266_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b5f5da4f-4e44-44db-836e-d414b822c9c9_isrForCrosstalkSources_871266_40.stderr
parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93771cdb7b6476f6b38bc46849c36470 with result from task 55
parsl.dataflow.dflow DEBUG: Task 55 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a4fb1ecb-351f-47cc-b98d-d5617b4bd01e_isrForCrosstalkSources_871266_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a4fb1ecb-351f-47cc-b98d-d5617b4bd01e_isrForCrosstalkSources_871266_7.stderr
parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 977fb01a8725eed7ce1839d8df61c28a with result from task 67
parsl.dataflow.dflow DEBUG: Task 67 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0abdb050-5818-4978-82cf-8e923675fcfe_isrForCrosstalkSources_871267_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0abdb050-5818-4978-82cf-8e923675fcfe_isrForCrosstalkSources_871267_43.stderr
parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a69547b45bd35c37e4c1d1b666e266d with result from task 59
parsl.dataflow.dflow DEBUG: Task 59 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/dd62aa76-ff1a-480a-a239-11ff3722aa28_isrForCrosstalkSources_871267_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/dd62aa76-ff1a-480a-a239-11ff3722aa28_isrForCrosstalkSources_871267_3.stderr
parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 42d9adda434279992a6417303fad7030 with result from task 63
parsl.dataflow.dflow DEBUG: Task 63 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c60f9697-eac3-4df6-9072-75a40d61e8d5_isrForCrosstalkSources_871265_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c60f9697-eac3-4df6-9072-75a40d61e8d5_isrForCrosstalkSources_871265_48.stderr
parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d024663ea23feef2c6fa0902480d1b9b with result from task 57
parsl.dataflow.dflow DEBUG: Task 57 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/50d64024-d417-40e4-b0a8-6553b5736776_isrForCrosstalkSources_871267_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/50d64024-d417-40e4-b0a8-6553b5736776_isrForCrosstalkSources_871267_35.stderr
parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7df1e1b154ad6f28506fada1a116fea9 with result from task 62
parsl.dataflow.dflow DEBUG: Task 62 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/cb332051-af61-4d52-ad9d-ec442ba1dff2_isrForCrosstalkSources_871265_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/cb332051-af61-4d52-ad9d-ec442ba1dff2_isrForCrosstalkSources_871265_62.stderr
parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 860e00ac238adea0f37156cd5a9b43e2 with result from task 76
parsl.dataflow.dflow DEBUG: Task 76 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e1c60f26-2d44-42fa-9708-853d828feece_isrForCrosstalkSources_871266_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e1c60f26-2d44-42fa-9708-853d828feece_isrForCrosstalkSources_871266_38.stderr
parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aae64dec77c40627bbe5603a2f4c4010 with result from task 58
parsl.dataflow.dflow DEBUG: Task 58 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d09be141-2f83-4d12-897b-394b7a8d09ec_isrForCrosstalkSources_871267_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d09be141-2f83-4d12-897b-394b7a8d09ec_isrForCrosstalkSources_871267_61.stderr
parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9689d6bfb9c25769a2d4d8bb247e468d with result from task 71
parsl.dataflow.dflow DEBUG: Task 71 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/51497508-babe-4cff-8280-cd2b614676c4_isrForCrosstalkSources_871267_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/51497508-babe-4cff-8280-cd2b614676c4_isrForCrosstalkSources_871267_58.stderr
parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 61b0fffa2a40f40e4416fb8264ea8045 with result from task 75
parsl.dataflow.dflow DEBUG: Task 75 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/48016698-1f46-4b08-b219-099bdd5f6a83_isrForCrosstalkSources_871265_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/48016698-1f46-4b08-b219-099bdd5f6a83_isrForCrosstalkSources_871265_43.stderr
parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc2e6d649ada1204a5822cf605763d2e with result from task 73
parsl.dataflow.dflow DEBUG: Task 73 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/69d6d115-61be-4d3b-8e10-74ba6e91aae1_isrForCrosstalkSources_871265_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/69d6d115-61be-4d3b-8e10-74ba6e91aae1_isrForCrosstalkSources_871265_37.stderr
parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00d231f34d907aed82043c006a9524f3 with result from task 66
parsl.dataflow.dflow DEBUG: Task 66 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e52ff88c-c359-4b21-a3f4-d40f2fa6317c_isrForCrosstalkSources_871267_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e52ff88c-c359-4b21-a3f4-d40f2fa6317c_isrForCrosstalkSources_871267_10.stderr
parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9e7ff7ed9e7a3859c4918ac4bf053350 with result from task 65
parsl.dataflow.dflow DEBUG: Task 65 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/6faa79fd-6df2-4d48-ba84-8312ff56fe5d_isrForCrosstalkSources_871265_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/6faa79fd-6df2-4d48-ba84-8312ff56fe5d_isrForCrosstalkSources_871265_13.stderr
parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 931e5fb738eff2f43c0d08973ecac321 with result from task 56
parsl.dataflow.dflow DEBUG: Task 56 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/74de8ee1-4ff0-4b10-8b4a-202c69e9f60c_isrForCrosstalkSources_871267_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/74de8ee1-4ff0-4b10-8b4a-202c69e9f60c_isrForCrosstalkSources_871267_25.stderr
parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dcf1e7fc225a6824b23c0dc9d81f0cb9 with result from task 77
parsl.dataflow.dflow DEBUG: Task 77 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3bf203d7-5c0e-47df-aa86-ca11eda4d5d3_isrForCrosstalkSources_871265_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3bf203d7-5c0e-47df-aa86-ca11eda4d5d3_isrForCrosstalkSources_871265_10.stderr
parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d572a328716a522bf1335a40e00cdfe0 with result from task 72
parsl.dataflow.dflow DEBUG: Task 72 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2a68200e-0e9a-4528-9858-ce447266e388_isrForCrosstalkSources_871265_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2a68200e-0e9a-4528-9858-ce447266e388_isrForCrosstalkSources_871265_20.stderr
parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 220fedf9369336a40bdd42c1516e3e6e with result from task 64
parsl.dataflow.dflow DEBUG: Task 64 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/84de0381-9d55-4c1d-872c-a7d07f7523b1_isrForCrosstalkSources_871265_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/84de0381-9d55-4c1d-872c-a7d07f7523b1_isrForCrosstalkSources_871265_9.stderr
parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5bcb53bc553b84662cc607f740f0f09 with result from task 74
parsl.dataflow.dflow DEBUG: Task 74 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1c84e2a1-ce95-4f7f-99c8-6d85ef4fb2d7_isrForCrosstalkSources_871267_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1c84e2a1-ce95-4f7f-99c8-6d85ef4fb2d7_isrForCrosstalkSources_871267_30.stderr
parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2970c2ebf59d2bd0cb18142d9776f40 with result from task 53
parsl.dataflow.dflow DEBUG: Task 53 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c6bc362d-a4e9-4ae0-9c1b-5ac94a7d7d72_isrForCrosstalkSources_871265_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c6bc362d-a4e9-4ae0-9c1b-5ac94a7d7d72_isrForCrosstalkSources_871265_39.stderr
parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0cb3172c075c8e4278a78b5832235171 with result from task 54
parsl.dataflow.dflow DEBUG: Task 54 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1ee7cc45-e718-45a3-9e6e-84d17fb37b87_isrForCrosstalkSources_871267_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1ee7cc45-e718-45a3-9e6e-84d17fb37b87_isrForCrosstalkSources_871267_15.stderr
2024-03-10 19:25:54 proc_lsst.multi:146 [INFO]  found job 96597 in provider local
proc_lsst.multi INFO: found job 96597 in provider local
2024-03-10 19:25:55 proc_lsst.multi:146 [INFO]  found job 16982456 in provider astro
proc_lsst.multi INFO: found job 16982456 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:55 proc_lsst.multi:146 [INFO]  found job 16982457 in provider astro
proc_lsst.multi INFO: found job 16982457 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37356ac67243bef399819d6e4f8f0344 with result from task 78
parsl.dataflow.dflow DEBUG: Task 78 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/258b8445-f8df-4cae-96d7-d58eaa8a9d65_isrForCrosstalkSources_871267_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/258b8445-f8df-4cae-96d7-d58eaa8a9d65_isrForCrosstalkSources_871267_46.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:55 proc_lsst.multi:146 [INFO]  found job 16982458 in provider astro
proc_lsst.multi INFO: found job 16982458 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:56 proc_lsst.multi:146 [INFO]  found job 16982459 in provider astro
proc_lsst.multi INFO: found job 16982459 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 96d45e0829ccdbcfa3824f105658257a with result from task 44
parsl.dataflow.dflow DEBUG: Task 44 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/af400b6e-c262-4895-a4bc-a59eb01ab5b8_isrForCrosstalkSources_871267_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/af400b6e-c262-4895-a4bc-a59eb01ab5b8_isrForCrosstalkSources_871267_38.stderr
parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c0affca01f280a332abbcc8c6d4ac7c0 with result from task 43
parsl.dataflow.dflow DEBUG: Task 43 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/33a35797-b2fe-4dc8-8196-177a94d65e3e_isrForCrosstalkSources_871267_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/33a35797-b2fe-4dc8-8196-177a94d65e3e_isrForCrosstalkSources_871267_56.stderr
parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a467a54c6b6ca6c133473292eeb0dd2 with result from task 47
parsl.dataflow.dflow DEBUG: Task 47 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8f67dd66-29f5-4a63-9d81-de067eee5b55_isrForCrosstalkSources_871265_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8f67dd66-29f5-4a63-9d81-de067eee5b55_isrForCrosstalkSources_871265_6.stderr
parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 84bcabaa05891415fa447d2c617dcf43 with result from task 42
parsl.dataflow.dflow DEBUG: Task 42 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fa5f097f-67bb-4061-8c53-981c0dc00337_isrForCrosstalkSources_871267_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fa5f097f-67bb-4061-8c53-981c0dc00337_isrForCrosstalkSources_871267_54.stderr
parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 27078ddda576c4b3d157311d5e71fbb3 with result from task 46
parsl.dataflow.dflow DEBUG: Task 46 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/4c99f6d5-b4ea-4de4-ab09-d0e267fbbf5c_isrForCrosstalkSources_871267_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/4c99f6d5-b4ea-4de4-ab09-d0e267fbbf5c_isrForCrosstalkSources_871267_50.stderr
parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 38d8be64c9f970a926ef61718f0533d9 with result from task 45
parsl.dataflow.dflow DEBUG: Task 45 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/ff806792-40f0-4fc0-a4e2-b72eec39e84c_isrForCrosstalkSources_871265_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/ff806792-40f0-4fc0-a4e2-b72eec39e84c_isrForCrosstalkSources_871265_27.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:56 proc_lsst.multi:146 [INFO]  found job 16982461 in provider astro
proc_lsst.multi INFO: found job 16982461 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:57 proc_lsst.multi:146 [INFO]  found job 16982462 in provider astro
proc_lsst.multi INFO: found job 16982462 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:57 proc_lsst.multi:146 [INFO]  found job 16982463 in provider astro
proc_lsst.multi INFO: found job 16982463 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:58 proc_lsst.multi:146 [INFO]  found job 16982464 in provider astro
proc_lsst.multi INFO: found job 16982464 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:58 proc_lsst.multi:146 [INFO]  found job 16982465 in provider astro
proc_lsst.multi INFO: found job 16982465 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d0f9db40ca71924994694e60bbce57c with result from task 79
parsl.dataflow.dflow DEBUG: Task 79 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/89b857d2-cc09-44b6-8449-2fddf717c90f_isrForCrosstalkSources_871265_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/89b857d2-cc09-44b6-8449-2fddf717c90f_isrForCrosstalkSources_871265_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:59 proc_lsst.multi:146 [INFO]  found job 16982466 in provider astro
proc_lsst.multi INFO: found job 16982466 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:25:59 proc_lsst.multi:146 [INFO]  found job 16982467 in provider astro
proc_lsst.multi INFO: found job 16982467 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:00 proc_lsst.multi:146 [INFO]  found job 16982468 in provider astro
proc_lsst.multi INFO: found job 16982468 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:00 proc_lsst.multi:146 [INFO]  found job 16982469 in provider astro
proc_lsst.multi INFO: found job 16982469 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:00 proc_lsst.multi:146 [INFO]  found job 16982470 in provider astro
proc_lsst.multi INFO: found job 16982470 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:01 proc_lsst.multi:146 [INFO]  found job 16982471 in provider astro
proc_lsst.multi INFO: found job 16982471 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:01 proc_lsst.multi:146 [INFO]  found job 16982472 in provider astro
proc_lsst.multi INFO: found job 16982472 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 117d31e5ac9077a14d50e46b45c83c9d with result from task 49
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.dflow DEBUG: Task 49 checkpointed
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:02 proc_lsst.multi:146 [INFO]  found job 16982473 in provider astro
proc_lsst.multi INFO: found job 16982473 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/868f1835-8025-4b5a-b263-833133c847a4_isrForCrosstalkSources_871267_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/868f1835-8025-4b5a-b263-833133c847a4_isrForCrosstalkSources_871267_9.stderr
parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67df0bbfdc62d958a379f6d5b8127f62 with result from task 48
parsl.dataflow.dflow DEBUG: Task 48 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/9f4d259f-b2cd-4413-942b-b4c93fa7f9d4_isrForCrosstalkSources_871267_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/9f4d259f-b2cd-4413-942b-b4c93fa7f9d4_isrForCrosstalkSources_871267_52.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:02 proc_lsst.multi:146 [INFO]  found job 16982474 in provider astro
proc_lsst.multi INFO: found job 16982474 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d98a4873e9bd80e7444da0edff021f24 with result from task 80
parsl.dataflow.dflow DEBUG: Task 80 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/91abcfc3-bf16-4307-ac74-ff97489cd88d_isrForCrosstalkSources_871265_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/91abcfc3-bf16-4307-ac74-ff97489cd88d_isrForCrosstalkSources_871265_33.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:03 proc_lsst.multi:146 [INFO]  found job 16982475 in provider astro
proc_lsst.multi INFO: found job 16982475 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:03 proc_lsst.multi:146 [INFO]  found job 16982476 in provider astro
proc_lsst.multi INFO: found job 16982476 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a0731eb708c7bbf37130a338e6c4cacf with result from task 85
parsl.dataflow.dflow DEBUG: Task 85 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5244671d-1dab-4841-a1f5-306ee06f68df_isrForCrosstalkSources_871265_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5244671d-1dab-4841-a1f5-306ee06f68df_isrForCrosstalkSources_871265_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:03 proc_lsst.multi:146 [INFO]  found job 16982477 in provider astro
proc_lsst.multi INFO: found job 16982477 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:04 proc_lsst.multi:146 [INFO]  found job 16982478 in provider astro
proc_lsst.multi INFO: found job 16982478 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eab85e04f0d9be022e6a4827ce2eeac4 with result from task 87
parsl.dataflow.dflow DEBUG: Task 87 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/d9386eb4-c10c-4840-bd3a-5034434e98cb_isrForCrosstalkSources_871266_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/d9386eb4-c10c-4840-bd3a-5034434e98cb_isrForCrosstalkSources_871266_33.stderr
parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1be86fd3ba1bde64f9dcee84c69ea1d with result from task 81
parsl.dataflow.dflow DEBUG: Task 81 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8e4afc23-f3f4-4d67-a78a-72b095e89190_isrForCrosstalkSources_871265_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/8e4afc23-f3f4-4d67-a78a-72b095e89190_isrForCrosstalkSources_871265_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:04 proc_lsst.multi:146 [INFO]  found job 16982479 in provider astro
proc_lsst.multi INFO: found job 16982479 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe49d77c0adc73289d1631ce9538d8a1 with result from task 88
parsl.dataflow.dflow DEBUG: Task 88 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/5f98a1bd-3726-4bd1-9b14-3aeffedee265_isrForCrosstalkSources_871267_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/5f98a1bd-3726-4bd1-9b14-3aeffedee265_isrForCrosstalkSources_871267_6.stderr
parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 400debad5c38bae28b31303d3f9f21dc with result from task 116
parsl.dataflow.dflow DEBUG: Task 116 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/8b3632c6-4d5a-44e3-9862-20a0dd829c13_isrForCrosstalkSources_871266_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/8b3632c6-4d5a-44e3-9862-20a0dd829c13_isrForCrosstalkSources_871266_35.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:05 proc_lsst.multi:146 [INFO]  found job 16982480 in provider astro
proc_lsst.multi INFO: found job 16982480 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:05 proc_lsst.multi:146 [INFO]  found job 16982481 in provider astro
proc_lsst.multi INFO: found job 16982481 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 423b1ac4b0be2a8de390d5e2d3c5736c with result from task 101
parsl.dataflow.dflow DEBUG: Task 101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/6640bd41-f59d-4570-81c4-67dd500a2b92_isrForCrosstalkSources_871265_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/6640bd41-f59d-4570-81c4-67dd500a2b92_isrForCrosstalkSources_871265_26.stderr
parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aee41afffaf9869a1f15603498cf474b with result from task 90
parsl.dataflow.dflow DEBUG: Task 90 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/c9f023f1-6b76-46b6-9eea-d95013c7bf2a_isrForCrosstalkSources_871267_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/c9f023f1-6b76-46b6-9eea-d95013c7bf2a_isrForCrosstalkSources_871267_8.stderr
parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0497981527dc88839fc13cfe0927386a with result from task 96
parsl.dataflow.dflow DEBUG: Task 96 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2f54e10e-a39c-495b-91f4-b8a0b40c043f_isrForCrosstalkSources_871265_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2f54e10e-a39c-495b-91f4-b8a0b40c043f_isrForCrosstalkSources_871265_7.stderr
parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 210262d3c45a50b63d8209ec18fca6f1 with result from task 107
parsl.dataflow.dflow DEBUG: Task 107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/931d22fc-d736-420f-8198-bc23fcf9269a_isrForCrosstalkSources_871267_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/931d22fc-d736-420f-8198-bc23fcf9269a_isrForCrosstalkSources_871267_1.stderr
parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d701238f27ddd8abac64a8cab980dee0 with result from task 97
parsl.dataflow.dflow DEBUG: Task 97 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1086f740-5b5e-48e6-b279-5db5205addee_isrForCrosstalkSources_871267_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/1086f740-5b5e-48e6-b279-5db5205addee_isrForCrosstalkSources_871267_60.stderr
parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8322409ac54bddf6e1ed2f16ab9fe0a3 with result from task 108
parsl.dataflow.dflow DEBUG: Task 108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5bb08627-b89e-4f67-a7f2-d30e838605e8_isrForCrosstalkSources_871265_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5bb08627-b89e-4f67-a7f2-d30e838605e8_isrForCrosstalkSources_871265_12.stderr
parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 224f20f2cf426326986344c621a0a391 with result from task 92
parsl.dataflow.dflow DEBUG: Task 92 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/cf1506f4-9d6b-4959-a598-eac156c6ea16_isrForCrosstalkSources_871267_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/cf1506f4-9d6b-4959-a598-eac156c6ea16_isrForCrosstalkSources_871267_48.stderr
parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c8cf9b3b2f2b2ee68df5ccf3a52e58f with result from task 100
parsl.dataflow.dflow DEBUG: Task 100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b0a70a19-6f24-4e99-aa32-8bed22246631_isrForCrosstalkSources_871265_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b0a70a19-6f24-4e99-aa32-8bed22246631_isrForCrosstalkSources_871265_38.stderr
parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3f3bd2ace79a9ca2211f5d1deac7c43b with result from task 91
parsl.dataflow.dflow DEBUG: Task 91 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/13b38b74-0308-498d-b47d-028b595e78a0_isrForCrosstalkSources_871265_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/13b38b74-0308-498d-b47d-028b595e78a0_isrForCrosstalkSources_871265_17.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:05 proc_lsst.multi:146 [INFO]  found job 16982482 in provider astro
proc_lsst.multi INFO: found job 16982482 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2a421f0e3f4ce51279d12896f026f16 with result from task 103
parsl.dataflow.dflow DEBUG: Task 103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/bbb65d4e-0779-49c1-b30a-34ac1e1a40f8_isrForCrosstalkSources_871266_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/bbb65d4e-0779-49c1-b30a-34ac1e1a40f8_isrForCrosstalkSources_871266_43.stderr
parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6cda6836ed68b0a876f248a46a6f158e with result from task 99
parsl.dataflow.dflow DEBUG: Task 99 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5cebd64f-e7a9-425f-b919-8509f5b2dc15_isrForCrosstalkSources_871266_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5cebd64f-e7a9-425f-b919-8509f5b2dc15_isrForCrosstalkSources_871266_53.stderr
parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 05506e0cf369372abaf19482e7f928be with result from task 102
parsl.dataflow.dflow DEBUG: Task 102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/455a6be3-3752-4ab9-b264-406a2fdb5554_isrForCrosstalkSources_871266_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/455a6be3-3752-4ab9-b264-406a2fdb5554_isrForCrosstalkSources_871266_51.stderr
parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 30459d3d64aa3cfc5f8bd0a6cb71479a with result from task 105
parsl.dataflow.dflow DEBUG: Task 105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/db606f06-2680-41cc-a8aa-0f842d567698_isrForCrosstalkSources_871267_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/db606f06-2680-41cc-a8aa-0f842d567698_isrForCrosstalkSources_871267_32.stderr
parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 024e0ae133d195630be50aca91c2b5af with result from task 109
parsl.dataflow.dflow DEBUG: Task 109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/1cbb62d2-c1d8-4a74-be42-a83d85233590_isrForCrosstalkSources_871266_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/1cbb62d2-c1d8-4a74-be42-a83d85233590_isrForCrosstalkSources_871266_2.stderr
parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a25062da52c911555adf48039a0f35e4 with result from task 98
parsl.dataflow.dflow DEBUG: Task 98 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a7cd218a-447a-4a30-aaa6-af20a23f697a_isrForCrosstalkSources_871266_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a7cd218a-447a-4a30-aaa6-af20a23f697a_isrForCrosstalkSources_871266_9.stderr
parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f1cb38932ebfccc7e4b8b5b06d90722 with result from task 82
parsl.dataflow.dflow DEBUG: Task 82 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/537bca6c-87eb-42cc-ab2e-269a30ecd077_isrForCrosstalkSources_871265_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/537bca6c-87eb-42cc-ab2e-269a30ecd077_isrForCrosstalkSources_871265_21.stderr
parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7f2a5501b10c4c8984c34e72d6a7b4 with result from task 83
parsl.dataflow.dflow DEBUG: Task 83 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5563fe49-11b4-4c38-a4ee-d7b5b7d4551d_isrForCrosstalkSources_871266_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5563fe49-11b4-4c38-a4ee-d7b5b7d4551d_isrForCrosstalkSources_871266_24.stderr
parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a842f4b5c66d68f4a8192a52dcce0eab with result from task 89
parsl.dataflow.dflow DEBUG: Task 89 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4a0af287-a0ae-455c-83d4-f1943eb61c15_isrForCrosstalkSources_871265_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4a0af287-a0ae-455c-83d4-f1943eb61c15_isrForCrosstalkSources_871265_1.stderr
parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ccbe1cb6adc971249984912966413cd8 with result from task 104
parsl.dataflow.dflow DEBUG: Task 104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b7c8b0a8-b2fd-48bf-994d-d965654a7f0b_isrForCrosstalkSources_871267_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b7c8b0a8-b2fd-48bf-994d-d965654a7f0b_isrForCrosstalkSources_871267_41.stderr
parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 809267a51ada5d2bf4dffc5cacc603c6 with result from task 110
parsl.dataflow.dflow DEBUG: Task 110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/27df0f72-8176-4736-b2c9-fb3fd82f6d82_isrForCrosstalkSources_871266_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/27df0f72-8176-4736-b2c9-fb3fd82f6d82_isrForCrosstalkSources_871266_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:06 proc_lsst.multi:146 [INFO]  found job 16982483 in provider astro
proc_lsst.multi INFO: found job 16982483 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 436cdec4ebcadd340c17369feb4378dd with result from task 93
parsl.dataflow.dflow DEBUG: Task 93 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e054228d-a6ad-444d-b039-b185d5baf072_isrForCrosstalkSources_871266_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e054228d-a6ad-444d-b039-b185d5baf072_isrForCrosstalkSources_871266_49.stderr
parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0dc6afc89e20022c1b79b7baa285a6c8 with result from task 84
parsl.dataflow.dflow DEBUG: Task 84 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/af087b70-5d90-4da2-b23d-f93117a4cdbf_isrForCrosstalkSources_871266_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/af087b70-5d90-4da2-b23d-f93117a4cdbf_isrForCrosstalkSources_871266_28.stderr
parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2e4b76bcec03536646632a68d93b8b7 with result from task 86
parsl.dataflow.dflow DEBUG: Task 86 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/1f6fbe36-5dd0-428f-9b56-58157fc7027f_isrForCrosstalkSources_871265_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/1f6fbe36-5dd0-428f-9b56-58157fc7027f_isrForCrosstalkSources_871265_61.stderr
parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dbdbc0289730be458d93a0d88c28c029 with result from task 94
parsl.dataflow.dflow DEBUG: Task 94 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a1dfe69e-9ca0-432b-92b1-bddc45460f94_isrForCrosstalkSources_871266_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a1dfe69e-9ca0-432b-92b1-bddc45460f94_isrForCrosstalkSources_871266_8.stderr
parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a448d147f69f13bb227d760edefb249 with result from task 111
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:06 proc_lsst.multi:146 [INFO]  found job 16982484 in provider astro
proc_lsst.multi INFO: found job 16982484 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow DEBUG: Task 111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/155404bd-a88b-46b3-bc6b-14b4f649b3a0_isrForCrosstalkSources_871266_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/155404bd-a88b-46b3-bc6b-14b4f649b3a0_isrForCrosstalkSources_871266_27.stderr
parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4544792b43d900361153f754fccfd704 with result from task 114
parsl.dataflow.dflow DEBUG: Task 114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/92c4d2a1-9a63-4c2b-99fa-b79be9dcd7cc_isrForCrosstalkSources_871266_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/92c4d2a1-9a63-4c2b-99fa-b79be9dcd7cc_isrForCrosstalkSources_871266_47.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:06 proc_lsst.multi:146 [INFO]  found job 16982485 in provider astro
proc_lsst.multi INFO: found job 16982485 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5ab54a16ea6a4221864a8a0ddd19d8d with result from task 113
parsl.dataflow.dflow DEBUG: Task 113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/94576fc4-e55b-4127-9a5e-31bd1448dbeb_isrForCrosstalkSources_871265_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/94576fc4-e55b-4127-9a5e-31bd1448dbeb_isrForCrosstalkSources_871265_54.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:07 proc_lsst.multi:146 [INFO]  found job 16982486 in provider astro
proc_lsst.multi INFO: found job 16982486 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:07 proc_lsst.multi:146 [INFO]  found job 16982487 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982487 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 79f6cced68af980fbf43d58342cac731 with result from task 106
parsl.dataflow.dflow DEBUG: Task 106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b466e922-a669-4734-be94-34b7403b3b66_isrForCrosstalkSources_871267_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b466e922-a669-4734-be94-34b7403b3b66_isrForCrosstalkSources_871267_22.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:07 proc_lsst.multi:146 [INFO]  found job 16982488 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982488 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b2e3fc057dfb69b96940b8fae322d027 with result from task 123
parsl.dataflow.dflow DEBUG: Task 123 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/947d9bcc-9c3d-49f9-a6fe-a556b9cbbb6b_isrForCrosstalkSources_871266_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/947d9bcc-9c3d-49f9-a6fe-a556b9cbbb6b_isrForCrosstalkSources_871266_61.stderr
parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 01ace307b47ccd14deddc53fe11a752e with result from task 120
parsl.dataflow.dflow DEBUG: Task 120 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/149e5493-d788-47b6-8375-ced13f896f9d_isrForCrosstalkSources_871266_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/149e5493-d788-47b6-8375-ced13f896f9d_isrForCrosstalkSources_871266_34.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:10 proc_lsst.multi:146 [INFO]  found job 16982489 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982489 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8ab006a2839ecedef33bb70d0c9e8265 with result from task 117
parsl.dataflow.dflow DEBUG: Task 117 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/d2cbcc0e-e43f-4d9d-a4ce-7985b6b39032_isrForCrosstalkSources_871265_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/d2cbcc0e-e43f-4d9d-a4ce-7985b6b39032_isrForCrosstalkSources_871265_24.stderr
parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 27835a72abd0daab7fbd86a79bec94a9 with result from task 95
parsl.dataflow.dflow DEBUG: Task 95 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0992a95f-40a0-45b2-aed5-a0d79c12ad85_isrForCrosstalkSources_871267_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0992a95f-40a0-45b2-aed5-a0d79c12ad85_isrForCrosstalkSources_871267_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:10 proc_lsst.multi:146 [INFO]  found job 16982490 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982490 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53f346ce080281a0f49bfef3238f8e68 with result from task 121
parsl.dataflow.dflow DEBUG: Task 121 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/199f01e9-e3a3-4d43-95b0-d404cca3c734_isrForCrosstalkSources_871266_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/199f01e9-e3a3-4d43-95b0-d404cca3c734_isrForCrosstalkSources_871266_22.stderr
parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 92399324e78471a8eb65489d11328d25 with result from task 118
parsl.dataflow.dflow DEBUG: Task 118 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b09c8eb4-9399-410e-81c0-619e035a9eb3_isrForCrosstalkSources_871267_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b09c8eb4-9399-410e-81c0-619e035a9eb3_isrForCrosstalkSources_871267_12.stderr
parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9bea8061bc34f37d825d597a851c541 with result from task 122
parsl.dataflow.dflow DEBUG: Task 122 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/f270b49f-7959-4091-b93c-4115f518ce91_isrForCrosstalkSources_871267_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/f270b49f-7959-4091-b93c-4115f518ce91_isrForCrosstalkSources_871267_49.stderr
parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 74a853701ebac6729706af89167f1645 with result from task 119
parsl.dataflow.dflow DEBUG: Task 119 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fe4a9b68-9b82-492a-b91e-eb9f742d9fbf_isrForCrosstalkSources_871267_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/fe4a9b68-9b82-492a-b91e-eb9f742d9fbf_isrForCrosstalkSources_871267_42.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:10 proc_lsst.multi:146 [INFO]  found job 16982491 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982491 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:11 proc_lsst.multi:146 [INFO]  found job 16982492 in provider escience_a40
proc_lsst.multi INFO: found job 16982492 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:11 proc_lsst.multi:146 [INFO]  found job 16982493 in provider escience_a40
proc_lsst.multi INFO: found job 16982493 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7e178f268c4e88728fb58e7da7f1ddf1 with result from task 112
parsl.dataflow.dflow DEBUG: Task 112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/d4e154f3-c05d-45fb-878d-f8c320c11935_isrForCrosstalkSources_871266_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/d4e154f3-c05d-45fb-878d-f8c320c11935_isrForCrosstalkSources_871266_30.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:12 proc_lsst.multi:146 [INFO]  found job 16982494 in provider escience_a40
proc_lsst.multi INFO: found job 16982494 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b529079a8e2eaac2855400c6b19bb382 with result from task 115
parsl.dataflow.dflow DEBUG: Task 115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/7278cb21-00ac-4168-a1ff-da2599fba5ee_isrForCrosstalkSources_871267_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/7278cb21-00ac-4168-a1ff-da2599fba5ee_isrForCrosstalkSources_871267_29.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:12 proc_lsst.multi:146 [INFO]  found job 16982495 in provider escience_a40
proc_lsst.multi INFO: found job 16982495 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:12 proc_lsst.multi:146 [INFO]  found job 16982496 in provider escience_a40
proc_lsst.multi INFO: found job 16982496 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:13 proc_lsst.multi:146 [INFO]  found job 16982497 in provider escience_a40
proc_lsst.multi INFO: found job 16982497 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:13 proc_lsst.multi:146 [INFO]  found job 16982498 in provider escience_a40
proc_lsst.multi INFO: found job 16982498 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:14 proc_lsst.multi:146 [INFO]  found job 16982499 in provider escience_a40
proc_lsst.multi INFO: found job 16982499 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 98b50018e928fef090d468931be24da7 with result from task 128
parsl.dataflow.dflow DEBUG: Task 128 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f09f8b52-d20d-4f04-81cf-6a6e08f8050e_isrForCrosstalkSources_871266_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f09f8b52-d20d-4f04-81cf-6a6e08f8050e_isrForCrosstalkSources_871266_6.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:14 proc_lsst.multi:146 [INFO]  found job 16982500 in provider escience_a40
proc_lsst.multi INFO: found job 16982500 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 94c12af85bdcc071558cfa83207426e2 with result from task 133
parsl.dataflow.dflow DEBUG: Task 133 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/072c1416-99f3-4b41-b468-ac9f27454d72_isrForCrosstalkSources_871267_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/072c1416-99f3-4b41-b468-ac9f27454d72_isrForCrosstalkSources_871267_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:14 proc_lsst.multi:146 [INFO]  found job 16982501 in provider escience_a40
proc_lsst.multi INFO: found job 16982501 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:15 proc_lsst.multi:146 [INFO]  found job 16982502 in provider escience_a40
proc_lsst.multi INFO: found job 16982502 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 555312106a5afac92508c0476debeed1 with result from task 127
parsl.dataflow.dflow DEBUG: Task 127 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/14d97276-fd43-4f46-96de-d837fa45bd35_isrForCrosstalkSources_871267_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/14d97276-fd43-4f46-96de-d837fa45bd35_isrForCrosstalkSources_871267_19.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 59
parsl.dataflow.strategy DEBUG: Executor multi has 59 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 59
parsl.dataflow.strategy DEBUG: Executor multi has 59 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33d2e886da381e874efaf3d569f76ac1 with result from task 126
parsl.dataflow.dflow DEBUG: Task 126 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/da535170-cfd0-4c79-8566-46091f589f65_isrForCrosstalkSources_871265_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/da535170-cfd0-4c79-8566-46091f589f65_isrForCrosstalkSources_871265_51.stderr
parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da957c7c5e2d13a7e6f27ff4b81b5f97 with result from task 138
parsl.dataflow.dflow DEBUG: Task 138 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/1758d504-2afc-4a89-8a2d-8466e63f5f39_isrForCrosstalkSources_871265_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/1758d504-2afc-4a89-8a2d-8466e63f5f39_isrForCrosstalkSources_871265_25.stderr
parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f74116eb2bc7fddbaad35d821cb1b476 with result from task 139
parsl.dataflow.dflow DEBUG: Task 139 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b771577c-2494-440b-adc2-267a42afc903_isrForCrosstalkSources_871267_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/b771577c-2494-440b-adc2-267a42afc903_isrForCrosstalkSources_871267_16.stderr
parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b456a90809b62182c61f139587550dc3 with result from task 143
parsl.dataflow.dflow DEBUG: Task 143 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/eb15782f-61ba-4110-bca3-70b6168b62d4_isrForCrosstalkSources_871266_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/eb15782f-61ba-4110-bca3-70b6168b62d4_isrForCrosstalkSources_871266_19.stderr
parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2f8b49dfa4a690353a7b1595373f329 with result from task 140
parsl.dataflow.dflow DEBUG: Task 140 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/dfe32577-59b4-4a21-9d1a-a0d5ac7faa4b_isrForCrosstalkSources_871266_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/dfe32577-59b4-4a21-9d1a-a0d5ac7faa4b_isrForCrosstalkSources_871266_15.stderr
parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f7c5208a94f6b1959a360f27deea2dc with result from task 145
parsl.dataflow.dflow DEBUG: Task 145 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a082ef0f-3891-4c42-979f-697c1a22035d_isrForCrosstalkSources_871266_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a082ef0f-3891-4c42-979f-697c1a22035d_isrForCrosstalkSources_871266_13.stderr
parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d9c712debb1a5daaa84cc538bd3b99e5 with result from task 146
parsl.dataflow.dflow DEBUG: Task 146 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0b47a4c4-8c4a-4935-bb13-63a1ccb67bed_isrForCrosstalkSources_871267_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0b47a4c4-8c4a-4935-bb13-63a1ccb67bed_isrForCrosstalkSources_871267_7.stderr
parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ef537479a7885f4b0d6aae6d236197b with result from task 135
parsl.dataflow.dflow DEBUG: Task 135 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/de65e774-d6a4-4753-acb6-6c3b5f75d0b1_isrForCrosstalkSources_871265_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/de65e774-d6a4-4753-acb6-6c3b5f75d0b1_isrForCrosstalkSources_871265_40.stderr
parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 523ec6b87be93e60cfe0ec646a1bf366 with result from task 136
parsl.dataflow.dflow DEBUG: Task 136 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e6f8e3f2-1508-4300-97e6-8adefc752d48_isrForCrosstalkSources_871266_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e6f8e3f2-1508-4300-97e6-8adefc752d48_isrForCrosstalkSources_871266_12.stderr
parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 685b1e338bc80bd7925ee06be1a19867 with result from task 152
parsl.dataflow.dflow DEBUG: Task 152 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/387c8d11-bc9f-40d0-a605-1b500ea0fb19_isrForCrosstalkSources_871266_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/387c8d11-bc9f-40d0-a605-1b500ea0fb19_isrForCrosstalkSources_871266_4.stderr
parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9d0208c8929584eb6b6f28fd9e4c702 with result from task 148
parsl.dataflow.dflow DEBUG: Task 148 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/ee17da96-dc19-4683-ab5a-f55c753d5345_isrForCrosstalkSources_871266_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/ee17da96-dc19-4683-ab5a-f55c753d5345_isrForCrosstalkSources_871266_52.stderr
parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a03c94268d033d6c4075c976f920c823 with result from task 158
parsl.dataflow.dflow DEBUG: Task 158 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/15c84f6b-371b-4061-a2eb-f812013a34a6_isrForCrosstalkSources_871267_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/15c84f6b-371b-4061-a2eb-f812013a34a6_isrForCrosstalkSources_871267_31.stderr
parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d62ec1e791c48ca836938113ac89a83 with result from task 162
parsl.dataflow.dflow DEBUG: Task 162 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3745997b-9d12-4531-9a59-d8c164af8626_isrForCrosstalkSources_871265_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3745997b-9d12-4531-9a59-d8c164af8626_isrForCrosstalkSources_871265_60.stderr
parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4de911fbb647da0f0ce9459eb98cdb8d with result from task 144
parsl.dataflow.dflow DEBUG: Task 144 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c1cf8330-9811-4146-9c13-ffe010a2952f_isrForCrosstalkSources_871265_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/c1cf8330-9811-4146-9c13-ffe010a2952f_isrForCrosstalkSources_871265_2.stderr
parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a3c00669c66761381e38598792b9b51 with result from task 141
parsl.dataflow.dflow DEBUG: Task 141 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2e1470f6-62e3-4281-a8d8-9b8e941dbf79_isrForCrosstalkSources_871265_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/2e1470f6-62e3-4281-a8d8-9b8e941dbf79_isrForCrosstalkSources_871265_36.stderr
parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b838fc2ab38f7d4ee1bca50f4cb9d3a with result from task 134
parsl.dataflow.dflow DEBUG: Task 134 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/42c2145e-fc23-4573-8e26-8820d7a97fd5_isrForCrosstalkSources_871265_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/42c2145e-fc23-4573-8e26-8820d7a97fd5_isrForCrosstalkSources_871265_30.stderr
parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9dceac1c028b53d7f7665d5c6a1d570b with result from task 157
parsl.dataflow.dflow DEBUG: Task 157 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/af734154-783b-46ce-83d8-a77f09309118_isrForCrosstalkSources_871267_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/af734154-783b-46ce-83d8-a77f09309118_isrForCrosstalkSources_871267_2.stderr
parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f846a4742c43381ac2848dceac8f1f35 with result from task 142
parsl.dataflow.dflow DEBUG: Task 142 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/45c93c2d-f179-48a6-b033-1ee3e0bf504b_isrForCrosstalkSources_871265_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/45c93c2d-f179-48a6-b033-1ee3e0bf504b_isrForCrosstalkSources_871265_41.stderr
parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 13ae4fd2888a32d123b8f7bb6a775a61 with result from task 159
parsl.dataflow.dflow DEBUG: Task 159 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/4bd0a408-641f-44bc-ba28-2b88e75b43c5_isrForCrosstalkSources_871267_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/4bd0a408-641f-44bc-ba28-2b88e75b43c5_isrForCrosstalkSources_871267_44.stderr
parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c26fdb82b7d568175af909dbaf78baf with result from task 131
parsl.dataflow.dflow DEBUG: Task 131 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f658b290-bc51-46f7-b6ad-ad6ce3437f7e_isrForCrosstalkSources_871266_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f658b290-bc51-46f7-b6ad-ad6ce3437f7e_isrForCrosstalkSources_871266_60.stderr
parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 65a14a6df5ccfd520c8d9a76a7a01ced with result from task 132
parsl.dataflow.dflow DEBUG: Task 132 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b0447118-d1a6-4398-ac91-6b9c10575e8b_isrForCrosstalkSources_871266_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b0447118-d1a6-4398-ac91-6b9c10575e8b_isrForCrosstalkSources_871266_20.stderr
parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bddd4eb0fa716f1d283ddad9ef1cd22f with result from task 154
parsl.dataflow.dflow DEBUG: Task 154 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/9e865543-9449-482b-96ac-2edbf6c24c2a_isrForCrosstalkSources_871265_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/9e865543-9449-482b-96ac-2edbf6c24c2a_isrForCrosstalkSources_871265_47.stderr
parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d89bdc573634f57623c01b15628f10b with result from task 130
parsl.dataflow.dflow DEBUG: Task 130 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3ab41666-d0a8-426a-8b91-b484f2bf120f_isrForCrosstalkSources_871265_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/3ab41666-d0a8-426a-8b91-b484f2bf120f_isrForCrosstalkSources_871265_58.stderr
parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52a0d1f239289edfa5974d06c2dc7543 with result from task 147
parsl.dataflow.dflow DEBUG: Task 147 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/73fd5ed7-b7e3-4ead-86bf-181e87f98215_isrForCrosstalkSources_871265_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/73fd5ed7-b7e3-4ead-86bf-181e87f98215_isrForCrosstalkSources_871265_52.stderr
parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eab1f30a8f632e7e034e3b93585847db with result from task 129
parsl.dataflow.dflow DEBUG: Task 129 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/15f0de64-dfec-4398-ab35-7c05878762a7_isrForCrosstalkSources_871265_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/15f0de64-dfec-4398-ab35-7c05878762a7_isrForCrosstalkSources_871265_53.stderr
parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ec8dea4ecdafa55123b7b5b928836d92 with result from task 137
parsl.dataflow.dflow DEBUG: Task 137 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/a23a44c0-e989-445b-bc6d-1276d87aa882_isrForCrosstalkSources_871265_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/a23a44c0-e989-445b-bc6d-1276d87aa882_isrForCrosstalkSources_871265_57.stderr
parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a54b144860c34ff87a840fad081c0c7a with result from task 153
parsl.dataflow.dflow DEBUG: Task 153 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b46d5046-37bb-43d7-a18a-48b62f763ec3_isrForCrosstalkSources_871265_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/b46d5046-37bb-43d7-a18a-48b62f763ec3_isrForCrosstalkSources_871265_31.stderr
parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0f0600d0429029b5b2fd2f65c569ab1 with result from task 160
parsl.dataflow.dflow DEBUG: Task 160 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5ab12821-ce48-4d51-b618-94f44ef69b1a_isrForCrosstalkSources_871265_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/5ab12821-ce48-4d51-b618-94f44ef69b1a_isrForCrosstalkSources_871265_50.stderr
parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95ec47f38bf048bf84314f4158de4a81 with result from task 161
parsl.dataflow.dflow DEBUG: Task 161 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/84ceeafb-0d07-4be5-a387-9628ae907c6f_isrForCrosstalkSources_871265_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/84ceeafb-0d07-4be5-a387-9628ae907c6f_isrForCrosstalkSources_871265_3.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 30
parsl.dataflow.strategy DEBUG: Executor multi has 30 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d7ae4b9a5e794200d025032d1beaa07 with result from task 151
parsl.dataflow.dflow DEBUG: Task 151 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e2580183-d4aa-4ae7-92fb-83f24713822e_isrForCrosstalkSources_871267_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/e2580183-d4aa-4ae7-92fb-83f24713822e_isrForCrosstalkSources_871267_13.stderr
parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d5d01bfa115eb89af0b73383153bfea8 with result from task 149
parsl.dataflow.dflow DEBUG: Task 149 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0b958f18-afae-47c0-b1b6-705c31f66395_isrForCrosstalkSources_871267_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/0b958f18-afae-47c0-b1b6-705c31f66395_isrForCrosstalkSources_871267_40.stderr
parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8efa3f046e2df99875a48ea77aeb8dee with result from task 150
parsl.dataflow.dflow DEBUG: Task 150 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f937b6b8-7134-4946-85b4-f827b58ce531_isrForCrosstalkSources_871266_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/f937b6b8-7134-4946-85b4-f827b58ce531_isrForCrosstalkSources_871266_3.stderr
parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f3fe79dcc394c5d6337849e06a02ed5c with result from task 155
parsl.dataflow.dflow DEBUG: Task 155 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/6e37e05d-9944-43c7-96c1-4bbdb6f72f5b_isrForCrosstalkSources_871267_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/6e37e05d-9944-43c7-96c1-4bbdb6f72f5b_isrForCrosstalkSources_871267_14.stderr
parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6cdd67396c4e279747b7f5b7622a826 with result from task 156
parsl.dataflow.dflow DEBUG: Task 156 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/02731c8a-de60-44b7-9b00-9fc84e62c3f8_isrForCrosstalkSources_871267_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/02731c8a-de60-44b7-9b00-9fc84e62c3f8_isrForCrosstalkSources_871267_51.stderr
parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9b8eabf3b90788e91543780f970fda1 with result from task 165
parsl.dataflow.dflow DEBUG: Task 165 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5b78c897-7a82-468b-ae11-3f015ef0c559_isrForCrosstalkSources_871266_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/5b78c897-7a82-468b-ae11-3f015ef0c559_isrForCrosstalkSources_871266_48.stderr
parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b1d54d907ec7b70a6c03c7a6703b9c2c with result from task 171
parsl.dataflow.dflow DEBUG: Task 171 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/fb76c9a1-7128-4319-99fc-f5fb2e9a08d2_isrForCrosstalkSources_871266_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/fb76c9a1-7128-4319-99fc-f5fb2e9a08d2_isrForCrosstalkSources_871266_41.stderr
parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 802a4a27b72d4cf2ebe52c8b9c8642c9 with result from task 170
parsl.dataflow.dflow DEBUG: Task 170 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/09289095-185b-4e1a-8509-4755fa9b1506_isrForCrosstalkSources_871265_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/09289095-185b-4e1a-8509-4755fa9b1506_isrForCrosstalkSources_871265_8.stderr
parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry be6dd4a0d16c691e32fd298812cf138a with result from task 173
parsl.dataflow.dflow DEBUG: Task 173 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/697d56fd-c7d2-4047-86cb-24674bb95b8f_isrForCrosstalkSources_871265_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/697d56fd-c7d2-4047-86cb-24674bb95b8f_isrForCrosstalkSources_871265_14.stderr
parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 230e21e33029c9f509f76732b0d1cec2 with result from task 164
parsl.dataflow.dflow DEBUG: Task 164 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/21b38c98-1e75-4644-bcba-50b5710fc99c_isrForCrosstalkSources_871266_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/21b38c98-1e75-4644-bcba-50b5710fc99c_isrForCrosstalkSources_871266_32.stderr
2024-03-10 19:26:25 proc_lsst.multi:146 [INFO]  found job 96597 in provider local
proc_lsst.multi INFO: found job 96597 in provider local
parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b13e6caa89149567b90da8243cda7190 with result from task 172
parsl.dataflow.dflow DEBUG: Task 172 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/ede6572d-50c8-4a21-9733-4ad274289c9a_isrForCrosstalkSources_871266_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/ede6572d-50c8-4a21-9733-4ad274289c9a_isrForCrosstalkSources_871266_26.stderr
2024-03-10 19:26:25 proc_lsst.multi:146 [INFO]  found job 16982456 in provider astro
proc_lsst.multi INFO: found job 16982456 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:26 proc_lsst.multi:146 [INFO]  found job 16982457 in provider astro
proc_lsst.multi INFO: found job 16982457 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 38864e0cf8346dffa1a6cd27020bce84 with result from task 163
parsl.dataflow.dflow DEBUG: Task 163 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/50eb28c2-eb1f-435b-9dee-45bd60d072d6_isrForCrosstalkSources_871267_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/50eb28c2-eb1f-435b-9dee-45bd60d072d6_isrForCrosstalkSources_871267_45.stderr
parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3008b1515b187097ac2667b30c3d3ecf with result from task 167
parsl.dataflow.dflow DEBUG: Task 167 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/42e51c68-307c-415d-b4c5-f8e3378e18d1_isrForCrosstalkSources_871267_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/42e51c68-307c-415d-b4c5-f8e3378e18d1_isrForCrosstalkSources_871267_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:26 proc_lsst.multi:146 [INFO]  found job 16982458 in provider astro
proc_lsst.multi INFO: found job 16982458 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 16bbd2ac4b626a4eacd50f9b2ed877ba with result from task 169
parsl.dataflow.dflow DEBUG: Task 169 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/eb5bd190-f164-4e22-8921-9d8e87afaa89_isrForCrosstalkSources_871265_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/eb5bd190-f164-4e22-8921-9d8e87afaa89_isrForCrosstalkSources_871265_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:26 proc_lsst.multi:146 [INFO]  found job 16982459 in provider astro
proc_lsst.multi INFO: found job 16982459 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:26 proc_lsst.multi:146 [INFO]  found job 16982461 in provider astro
proc_lsst.multi INFO: found job 16982461 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:26 proc_lsst.multi:146 [INFO]  found job 16982462 in provider astro
proc_lsst.multi INFO: found job 16982462 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:27 proc_lsst.multi:146 [INFO]  found job 16982463 in provider astro
proc_lsst.multi INFO: found job 16982463 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78754eb925714e7e8e94c63634dc7c56 with result from task 166
parsl.dataflow.dflow DEBUG: Task 166 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/20fef37a-5fee-4607-a02c-86f3bff08d8a_isrForCrosstalkSources_871266_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/20fef37a-5fee-4607-a02c-86f3bff08d8a_isrForCrosstalkSources_871266_37.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:27 proc_lsst.multi:146 [INFO]  found job 16982464 in provider astro
proc_lsst.multi INFO: found job 16982464 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2399c0e95b1f1af8ec4028014df8c316 with result from task 176
parsl.dataflow.dflow DEBUG: Task 176 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/434fd797-b9c9-4a8d-928c-fbac00a7e56e_isrForCrosstalkSources_871266_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/434fd797-b9c9-4a8d-928c-fbac00a7e56e_isrForCrosstalkSources_871266_56.stderr
parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b575099d0303116ec01df6399a76c9da with result from task 179
parsl.dataflow.dflow DEBUG: Task 179 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/eafdc984-ea75-4ac9-a843-c3b30322b966_isrForCrosstalkSources_871266_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/eafdc984-ea75-4ac9-a843-c3b30322b966_isrForCrosstalkSources_871266_31.stderr
parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6be8b5ffcaac37372321f42ffd73bf92 with result from task 178
parsl.dataflow.dflow DEBUG: Task 178 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d8b40d8f-1f85-4a95-933b-e1b3fc130508_isrForCrosstalkSources_871267_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/d8b40d8f-1f85-4a95-933b-e1b3fc130508_isrForCrosstalkSources_871267_5.stderr
parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry afc993c3a50b33838b7df577521e792d with result from task 168
parsl.dataflow.dflow DEBUG: Task 168 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/7f6b9fae-5d71-4ca3-b1aa-2dc0872c2a9a_isrForCrosstalkSources_871266_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/7f6b9fae-5d71-4ca3-b1aa-2dc0872c2a9a_isrForCrosstalkSources_871266_59.stderr
parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 022ba1882b69068152d4863aa7b1f071 with result from task 177
parsl.dataflow.dflow DEBUG: Task 177 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/92918767-dab7-46b0-b552-f922cb1ea242_isrForCrosstalkSources_871267_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/92918767-dab7-46b0-b552-f922cb1ea242_isrForCrosstalkSources_871267_36.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:27 proc_lsst.multi:146 [INFO]  found job 16982465 in provider astro
proc_lsst.multi INFO: found job 16982465 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 60bbc05daccd9054faaf35027615ea52 with result from task 180
parsl.dataflow.dflow DEBUG: Task 180 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a2af5d4e-6858-4a1c-8a5e-e8f6ba50f6e2_isrForCrosstalkSources_871266_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/a2af5d4e-6858-4a1c-8a5e-e8f6ba50f6e2_isrForCrosstalkSources_871266_17.stderr
parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry af7905d4e874893e43b2a15165c28c18 with result from task 181
parsl.dataflow.dflow DEBUG: Task 181 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b6357928-134a-4745-8e2f-804691e17f97_isrForCrosstalkSources_871266_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/b6357928-134a-4745-8e2f-804691e17f97_isrForCrosstalkSources_871266_21.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:27 proc_lsst.multi:146 [INFO]  found job 16982466 in provider astro
proc_lsst.multi INFO: found job 16982466 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 425c1388ef9e14bbc9a66cc447f08879 with result from task 182
parsl.dataflow.dflow DEBUG: Task 182 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/19e4c572-a054-489f-83f2-d0113ee0e0f4_isrForCrosstalkSources_871265_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/19e4c572-a054-489f-83f2-d0113ee0e0f4_isrForCrosstalkSources_871265_56.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:27 proc_lsst.multi:146 [INFO]  found job 16982467 in provider astro
proc_lsst.multi INFO: found job 16982467 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aacf9907652d14ff3c870e654e958379 with result from task 185
parsl.dataflow.dflow DEBUG: Task 185 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/07a3e405-57a4-453f-bcf3-4b835f0c4bfc_isrForCrosstalkSources_871266_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/07a3e405-57a4-453f-bcf3-4b835f0c4bfc_isrForCrosstalkSources_871266_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:28 proc_lsst.multi:146 [INFO]  found job 16982468 in provider astro
proc_lsst.multi INFO: found job 16982468 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 49ea08bd1d6bfa7e8d426e6d4ffc74d9 with result from task 183
parsl.dataflow.dflow DEBUG: Task 183 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e5ae510c-1519-4ee1-ada9-00602996fcb1_isrForCrosstalkSources_871266_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/e5ae510c-1519-4ee1-ada9-00602996fcb1_isrForCrosstalkSources_871266_11.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:28 proc_lsst.multi:146 [INFO]  found job 16982469 in provider astro
proc_lsst.multi INFO: found job 16982469 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 212217bdb91ed28d456b32c452f6cab2 with result from task 184
parsl.dataflow.dflow DEBUG: Task 184 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/0fa4fced-5198-44bd-bd73-cb3d8828cc07_isrForCrosstalkSources_871265_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/0fa4fced-5198-44bd-bd73-cb3d8828cc07_isrForCrosstalkSources_871265_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:28 proc_lsst.multi:146 [INFO]  found job 16982470 in provider astro
proc_lsst.multi INFO: found job 16982470 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:28 proc_lsst.multi:146 [INFO]  found job 16982471 in provider astro
proc_lsst.multi INFO: found job 16982471 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5ad3c735d92324532478ebbf3fab852 with result from task 174
parsl.dataflow.dflow DEBUG: Task 174 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/2b60eeba-e62c-4fb5-a1c2-b14996e0ca31_isrForCrosstalkSources_871267_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871267/2b60eeba-e62c-4fb5-a1c2-b14996e0ca31_isrForCrosstalkSources_871267_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:28 proc_lsst.multi:146 [INFO]  found job 16982472 in provider astro
proc_lsst.multi INFO: found job 16982472 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:29 proc_lsst.multi:146 [INFO]  found job 16982473 in provider astro
proc_lsst.multi INFO: found job 16982473 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:29 proc_lsst.multi:146 [INFO]  found job 16982474 in provider astro
proc_lsst.multi INFO: found job 16982474 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:29 proc_lsst.multi:146 [INFO]  found job 16982475 in provider astro
proc_lsst.multi INFO: found job 16982475 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:29 proc_lsst.multi:146 [INFO]  found job 16982476 in provider astro
proc_lsst.multi INFO: found job 16982476 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:29 proc_lsst.multi:146 [INFO]  found job 16982477 in provider astro
proc_lsst.multi INFO: found job 16982477 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:29 proc_lsst.multi:146 [INFO]  found job 16982478 in provider astro
proc_lsst.multi INFO: found job 16982478 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:30 proc_lsst.multi:146 [INFO]  found job 16982479 in provider astro
proc_lsst.multi INFO: found job 16982479 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:30 proc_lsst.multi:146 [INFO]  found job 16982480 in provider astro
proc_lsst.multi INFO: found job 16982480 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:30 proc_lsst.multi:146 [INFO]  found job 16982481 in provider astro
proc_lsst.multi INFO: found job 16982481 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:30 proc_lsst.multi:146 [INFO]  found job 16982482 in provider astro
proc_lsst.multi INFO: found job 16982482 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:31 proc_lsst.multi:146 [INFO]  found job 16982483 in provider astro
proc_lsst.multi INFO: found job 16982483 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7fa57c22b2928eda53378e3a9d85986 with result from task 175
parsl.dataflow.dflow DEBUG: Task 175 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/949f7096-a6d1-4e9f-9dbc-601740c438c8_isrForCrosstalkSources_871266_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871266/949f7096-a6d1-4e9f-9dbc-601740c438c8_isrForCrosstalkSources_871266_18.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:31 proc_lsst.multi:146 [INFO]  found job 16982484 in provider astro
proc_lsst.multi INFO: found job 16982484 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:31 proc_lsst.multi:146 [INFO]  found job 16982485 in provider astro
proc_lsst.multi INFO: found job 16982485 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:31 proc_lsst.multi:146 [INFO]  found job 16982486 in provider astro
proc_lsst.multi INFO: found job 16982486 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:31 proc_lsst.multi:146 [INFO]  found job 16982487 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982487 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:32 proc_lsst.multi:146 [INFO]  found job 16982488 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982488 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:32 proc_lsst.multi:146 [INFO]  found job 16982489 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982489 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:32 proc_lsst.multi:146 [INFO]  found job 16982490 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982490 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:32 proc_lsst.multi:146 [INFO]  found job 16982491 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982491 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:32 proc_lsst.multi:146 [INFO]  found job 16982492 in provider escience_a40
proc_lsst.multi INFO: found job 16982492 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:32 proc_lsst.multi:146 [INFO]  found job 16982493 in provider escience_a40
proc_lsst.multi INFO: found job 16982493 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:33 proc_lsst.multi:146 [INFO]  found job 16982494 in provider escience_a40
proc_lsst.multi INFO: found job 16982494 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:33 proc_lsst.multi:146 [INFO]  found job 16982495 in provider escience_a40
proc_lsst.multi INFO: found job 16982495 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:33 proc_lsst.multi:146 [INFO]  found job 16982496 in provider escience_a40
proc_lsst.multi INFO: found job 16982496 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:33 proc_lsst.multi:146 [INFO]  found job 16982497 in provider escience_a40
proc_lsst.multi INFO: found job 16982497 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:33 proc_lsst.multi:146 [INFO]  found job 16982498 in provider escience_a40
proc_lsst.multi INFO: found job 16982498 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:34 proc_lsst.multi:146 [INFO]  found job 16982499 in provider escience_a40
proc_lsst.multi INFO: found job 16982499 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:34 proc_lsst.multi:146 [INFO]  found job 16982500 in provider escience_a40
proc_lsst.multi INFO: found job 16982500 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:34 proc_lsst.multi:146 [INFO]  found job 16982501 in provider escience_a40
proc_lsst.multi INFO: found job 16982501 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:26:34 proc_lsst.multi:146 [INFO]  found job 16982502 in provider escience_a40
proc_lsst.multi INFO: found job 16982502 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
2024-03-10 19:26:59 proc_lsst.multi:146 [INFO]  found job 96597 in provider local
proc_lsst.multi INFO: found job 96597 in provider local
2024-03-10 19:26:59 proc_lsst.multi:146 [INFO]  found job 16982456 in provider astro
proc_lsst.multi INFO: found job 16982456 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:00 proc_lsst.multi:146 [INFO]  found job 16982457 in provider astro
proc_lsst.multi INFO: found job 16982457 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:00 proc_lsst.multi:146 [INFO]  found job 16982458 in provider astro
proc_lsst.multi INFO: found job 16982458 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:00 proc_lsst.multi:146 [INFO]  found job 16982459 in provider astro
proc_lsst.multi INFO: found job 16982459 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:00 proc_lsst.multi:146 [INFO]  found job 16982461 in provider astro
proc_lsst.multi INFO: found job 16982461 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:00 proc_lsst.multi:146 [INFO]  found job 16982462 in provider astro
proc_lsst.multi INFO: found job 16982462 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:01 proc_lsst.multi:146 [INFO]  found job 16982463 in provider astro
proc_lsst.multi INFO: found job 16982463 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:01 proc_lsst.multi:146 [INFO]  found job 16982464 in provider astro
proc_lsst.multi INFO: found job 16982464 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:01 proc_lsst.multi:146 [INFO]  found job 16982465 in provider astro
proc_lsst.multi INFO: found job 16982465 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:01 proc_lsst.multi:146 [INFO]  found job 16982466 in provider astro
proc_lsst.multi INFO: found job 16982466 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:02 proc_lsst.multi:146 [INFO]  found job 16982467 in provider astro
proc_lsst.multi INFO: found job 16982467 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:02 proc_lsst.multi:146 [INFO]  found job 16982468 in provider astro
proc_lsst.multi INFO: found job 16982468 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:02 proc_lsst.multi:146 [INFO]  found job 16982469 in provider astro
proc_lsst.multi INFO: found job 16982469 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:02 proc_lsst.multi:146 [INFO]  found job 16982470 in provider astro
proc_lsst.multi INFO: found job 16982470 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:02 proc_lsst.multi:146 [INFO]  found job 16982471 in provider astro
proc_lsst.multi INFO: found job 16982471 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:03 proc_lsst.multi:146 [INFO]  found job 16982472 in provider astro
proc_lsst.multi INFO: found job 16982472 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:03 proc_lsst.multi:146 [INFO]  found job 16982473 in provider astro
proc_lsst.multi INFO: found job 16982473 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:03 proc_lsst.multi:146 [INFO]  found job 16982474 in provider astro
proc_lsst.multi INFO: found job 16982474 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:03 proc_lsst.multi:146 [INFO]  found job 16982475 in provider astro
proc_lsst.multi INFO: found job 16982475 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:03 proc_lsst.multi:146 [INFO]  found job 16982476 in provider astro
proc_lsst.multi INFO: found job 16982476 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:04 proc_lsst.multi:146 [INFO]  found job 16982477 in provider astro
proc_lsst.multi INFO: found job 16982477 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:04 proc_lsst.multi:146 [INFO]  found job 16982478 in provider astro
proc_lsst.multi INFO: found job 16982478 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:04 proc_lsst.multi:146 [INFO]  found job 16982479 in provider astro
proc_lsst.multi INFO: found job 16982479 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:04 proc_lsst.multi:146 [INFO]  found job 16982480 in provider astro
proc_lsst.multi INFO: found job 16982480 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:04 proc_lsst.multi:146 [INFO]  found job 16982481 in provider astro
proc_lsst.multi INFO: found job 16982481 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:05 proc_lsst.multi:146 [INFO]  found job 16982482 in provider astro
proc_lsst.multi INFO: found job 16982482 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:05 proc_lsst.multi:146 [INFO]  found job 16982483 in provider astro
proc_lsst.multi INFO: found job 16982483 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:05 proc_lsst.multi:146 [INFO]  found job 16982484 in provider astro
proc_lsst.multi INFO: found job 16982484 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:05 proc_lsst.multi:146 [INFO]  found job 16982485 in provider astro
proc_lsst.multi INFO: found job 16982485 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:05 proc_lsst.multi:146 [INFO]  found job 16982486 in provider astro
proc_lsst.multi INFO: found job 16982486 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:06 proc_lsst.multi:146 [INFO]  found job 16982487 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982487 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:06 proc_lsst.multi:146 [INFO]  found job 16982488 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982488 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:06 proc_lsst.multi:146 [INFO]  found job 16982489 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982489 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:06 proc_lsst.multi:146 [INFO]  found job 16982490 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982490 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:06 proc_lsst.multi:146 [INFO]  found job 16982491 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982491 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:07 proc_lsst.multi:146 [INFO]  found job 16982492 in provider escience_a40
proc_lsst.multi INFO: found job 16982492 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:07 proc_lsst.multi:146 [INFO]  found job 16982493 in provider escience_a40
proc_lsst.multi INFO: found job 16982493 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:07 proc_lsst.multi:146 [INFO]  found job 16982494 in provider escience_a40
proc_lsst.multi INFO: found job 16982494 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:07 proc_lsst.multi:146 [INFO]  found job 16982495 in provider escience_a40
proc_lsst.multi INFO: found job 16982495 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:07 proc_lsst.multi:146 [INFO]  found job 16982496 in provider escience_a40
proc_lsst.multi INFO: found job 16982496 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:07 proc_lsst.multi:146 [INFO]  found job 16982497 in provider escience_a40
proc_lsst.multi INFO: found job 16982497 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:08 proc_lsst.multi:146 [INFO]  found job 16982498 in provider escience_a40
proc_lsst.multi INFO: found job 16982498 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:08 proc_lsst.multi:146 [INFO]  found job 16982499 in provider escience_a40
proc_lsst.multi INFO: found job 16982499 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:08 proc_lsst.multi:146 [INFO]  found job 16982500 in provider escience_a40
proc_lsst.multi INFO: found job 16982500 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:08 proc_lsst.multi:146 [INFO]  found job 16982501 in provider escience_a40
proc_lsst.multi INFO: found job 16982501 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:08 proc_lsst.multi:146 [INFO]  found job 16982502 in provider escience_a40
proc_lsst.multi INFO: found job 16982502 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
2024-03-10 19:27:34 proc_lsst.multi:146 [INFO]  found job 96597 in provider local
proc_lsst.multi INFO: found job 96597 in provider local
2024-03-10 19:27:34 proc_lsst.multi:146 [INFO]  found job 16982456 in provider astro
proc_lsst.multi INFO: found job 16982456 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:34 proc_lsst.multi:146 [INFO]  found job 16982457 in provider astro
proc_lsst.multi INFO: found job 16982457 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:34 proc_lsst.multi:146 [INFO]  found job 16982458 in provider astro
proc_lsst.multi INFO: found job 16982458 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:34 proc_lsst.multi:146 [INFO]  found job 16982459 in provider astro
proc_lsst.multi INFO: found job 16982459 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:34 proc_lsst.multi:146 [INFO]  found job 16982461 in provider astro
proc_lsst.multi INFO: found job 16982461 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:35 proc_lsst.multi:146 [INFO]  found job 16982462 in provider astro
proc_lsst.multi INFO: found job 16982462 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:35 proc_lsst.multi:146 [INFO]  found job 16982463 in provider astro
proc_lsst.multi INFO: found job 16982463 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:35 proc_lsst.multi:146 [INFO]  found job 16982464 in provider astro
proc_lsst.multi INFO: found job 16982464 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:35 proc_lsst.multi:146 [INFO]  found job 16982465 in provider astro
proc_lsst.multi INFO: found job 16982465 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:35 proc_lsst.multi:146 [INFO]  found job 16982466 in provider astro
proc_lsst.multi INFO: found job 16982466 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:36 proc_lsst.multi:146 [INFO]  found job 16982467 in provider astro
proc_lsst.multi INFO: found job 16982467 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:36 proc_lsst.multi:146 [INFO]  found job 16982468 in provider astro
proc_lsst.multi INFO: found job 16982468 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:36 proc_lsst.multi:146 [INFO]  found job 16982469 in provider astro
proc_lsst.multi INFO: found job 16982469 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:36 proc_lsst.multi:146 [INFO]  found job 16982470 in provider astro
proc_lsst.multi INFO: found job 16982470 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:36 proc_lsst.multi:146 [INFO]  found job 16982471 in provider astro
proc_lsst.multi INFO: found job 16982471 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:37 proc_lsst.multi:146 [INFO]  found job 16982472 in provider astro
proc_lsst.multi INFO: found job 16982472 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:37 proc_lsst.multi:146 [INFO]  found job 16982473 in provider astro
proc_lsst.multi INFO: found job 16982473 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:37 proc_lsst.multi:146 [INFO]  found job 16982474 in provider astro
proc_lsst.multi INFO: found job 16982474 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:37 proc_lsst.multi:146 [INFO]  found job 16982475 in provider astro
proc_lsst.multi INFO: found job 16982475 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:37 proc_lsst.multi:146 [INFO]  found job 16982476 in provider astro
proc_lsst.multi INFO: found job 16982476 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:38 proc_lsst.multi:146 [INFO]  found job 16982477 in provider astro
proc_lsst.multi INFO: found job 16982477 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:38 proc_lsst.multi:146 [INFO]  found job 16982478 in provider astro
proc_lsst.multi INFO: found job 16982478 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:38 proc_lsst.multi:146 [INFO]  found job 16982479 in provider astro
proc_lsst.multi INFO: found job 16982479 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:38 proc_lsst.multi:146 [INFO]  found job 16982480 in provider astro
proc_lsst.multi INFO: found job 16982480 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:38 proc_lsst.multi:146 [INFO]  found job 16982481 in provider astro
proc_lsst.multi INFO: found job 16982481 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:39 proc_lsst.multi:146 [INFO]  found job 16982482 in provider astro
proc_lsst.multi INFO: found job 16982482 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:39 proc_lsst.multi:146 [INFO]  found job 16982483 in provider astro
proc_lsst.multi INFO: found job 16982483 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:39 proc_lsst.multi:146 [INFO]  found job 16982484 in provider astro
proc_lsst.multi INFO: found job 16982484 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:39 proc_lsst.multi:146 [INFO]  found job 16982485 in provider astro
proc_lsst.multi INFO: found job 16982485 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:39 proc_lsst.multi:146 [INFO]  found job 16982486 in provider astro
proc_lsst.multi INFO: found job 16982486 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982456,16982457,16982458,16982459,16982461,16982462,16982463,16982464,16982465,16982466,16982467,16982468,16982469,16982470,16982471,16982472,16982473,16982474,16982475,16982476,16982477,16982478,16982479,16982480,16982481,16982482,16982483,16982484,16982485,16982486'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982486 R
16982476 R
16982477 R
16982478 R
16982479 R
16982480 R
16982481 R
16982482 R
16982483 R
16982484 R
16982485 R
16982470 R
16982471 R
16982472 R
16982473 R
16982474 R
16982475 R
16982464 R
16982465 R
16982466 R
16982467 R
16982468 R
16982469 R
16982463 R
16982458 R
16982459 R
16982461 R
16982462 R
16982456 R
16982457 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982486 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982476 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982477 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982478 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982479 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982480 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982481 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982482 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982483 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982484 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982485 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982470 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982471 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982472 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982473 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982474 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982475 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982464 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982465 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982466 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982467 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982468 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982469 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982463 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982458 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982459 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982461 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982462 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982456 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982457 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:40 proc_lsst.multi:146 [INFO]  found job 16982487 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982487 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:40 proc_lsst.multi:146 [INFO]  found job 16982488 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982488 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:40 proc_lsst.multi:146 [INFO]  found job 16982489 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982489 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:40 proc_lsst.multi:146 [INFO]  found job 16982490 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982490 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:40 proc_lsst.multi:146 [INFO]  found job 16982491 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982491 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982487,16982488,16982489,16982490,16982491'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982487 R
16982488 R
16982489 R
16982490 R
16982491 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982487 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982488 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982489 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982490 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982491 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:40 proc_lsst.multi:146 [INFO]  found job 16982492 in provider escience_a40
proc_lsst.multi INFO: found job 16982492 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:41 proc_lsst.multi:146 [INFO]  found job 16982493 in provider escience_a40
proc_lsst.multi INFO: found job 16982493 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:41 proc_lsst.multi:146 [INFO]  found job 16982494 in provider escience_a40
proc_lsst.multi INFO: found job 16982494 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:41 proc_lsst.multi:146 [INFO]  found job 16982495 in provider escience_a40
proc_lsst.multi INFO: found job 16982495 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:41 proc_lsst.multi:146 [INFO]  found job 16982496 in provider escience_a40
proc_lsst.multi INFO: found job 16982496 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:41 proc_lsst.multi:146 [INFO]  found job 16982497 in provider escience_a40
proc_lsst.multi INFO: found job 16982497 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:42 proc_lsst.multi:146 [INFO]  found job 16982498 in provider escience_a40
proc_lsst.multi INFO: found job 16982498 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:42 proc_lsst.multi:146 [INFO]  found job 16982499 in provider escience_a40
proc_lsst.multi INFO: found job 16982499 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:42 proc_lsst.multi:146 [INFO]  found job 16982500 in provider escience_a40
proc_lsst.multi INFO: found job 16982500 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:42 proc_lsst.multi:146 [INFO]  found job 16982501 in provider escience_a40
proc_lsst.multi INFO: found job 16982501 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
2024-03-10 19:27:42 proc_lsst.multi:146 [INFO]  found job 16982502 in provider escience_a40
proc_lsst.multi INFO: found job 16982502 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16982492,16982493,16982494,16982495,16982496,16982497,16982498,16982499,16982500,16982501,16982502'
parsl.providers.slurm.slurm DEBUG: squeue returned 16982502 R
16982498 R
16982499 R
16982500 R
16982501 R
16982493 R
16982494 R
16982495 R
16982496 R
16982497 R
16982492 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 16982502 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982498 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982499 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982500 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982501 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982493 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982494 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982495 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982496 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982497 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 16982492 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2
parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23352583482064
parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b8ffd35285c3288fb2f7a8139397b3b with result from task 125
parsl.dataflow.dflow DEBUG: Task 125 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4b9b0c85-027c-4fe4-ab86-e271f7a76429_isrForCrosstalkSources_871265_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/4b9b0c85-027c-4fe4-ab86-e271f7a76429_isrForCrosstalkSources_871265_19.stderr
parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 473ef45ca01a37c841347eeda5d4304f with result from task 124
parsl.dataflow.dflow INFO: DFK cleanup initiated
parsl.dataflow.dflow INFO: Summary of tasks in DFK:
parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0
parsl.dataflow.dflow INFO: Tasks in state States.pending: 0
parsl.dataflow.dflow DEBUG: Task 124 checkpointed
parsl.dataflow.dflow INFO: Tasks in state States.running: 0
parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 186
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Tasks in state States.failed: 0
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/86832478-9781-45e2-b3ac-b73a24e5a627_isrForCrosstalkSources_871265_28.stdout
parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/submit/DEEP/20190707/B0b/science#step0/20240310T192404Z/logs/isrForCrosstalkSources/871265/86832478-9781-45e2-b3ac-b73a24e5a627_isrForCrosstalkSources_871265_28.stderr
parsl.dataflow.dflow INFO: Tasks in state States.launched: 0
parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0
parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0
parsl.dataflow.dflow INFO: Tasks in state States.joining: 0
parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0
parsl.dataflow.dflow INFO: End of summary
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Closing job status poller
parsl.dataflow.dflow INFO: Terminated job status poller
parsl.dataflow.dflow INFO: Scaling in and shutting down executors
parsl.dataflow.dflow INFO: Scaling in executor multi
parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=47, block_ids=[]
parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 47 blocks
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b0900db210a5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b0900db210a5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3888f2d38c80
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3888f2d38c80
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7eb4c081716e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7eb4c081716e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 78da5f72ffaf
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 78da5f72ffaf
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 68bbaee9d627
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 68bbaee9d627
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 77d456fb4642
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 77d456fb4642
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 34ec9bee38f0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 34ec9bee38f0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 075ff9d00654
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 075ff9d00654
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3f5a4e6d341f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3f5a4e6d341f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 34f6ecc1cc6b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 34f6ecc1cc6b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d150aab50ce5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d150aab50ce5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3a63590f6bd1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3a63590f6bd1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 42bd4914d8ac
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 42bd4914d8ac
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a91d85d75eef
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a91d85d75eef
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dbf4a9b913ea
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dbf4a9b913ea
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c3e3c65a6240
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c3e3c65a6240
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 752df27b5cf7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 752df27b5cf7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e1c3a26aaa8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e1c3a26aaa8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8a7009783c37
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8a7009783c37
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 99ece20ad971
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 99ece20ad971
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eda155ea96f1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eda155ea96f1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bdbbe1c66a0f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bdbbe1c66a0f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 235892bd5216
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 235892bd5216
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b3ea2dc14b00
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b3ea2dc14b00
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cea177044c42
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cea177044c42
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 430ac6c7b397
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 430ac6c7b397
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd9c5bdfb2b2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd9c5bdfb2b2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 335187ea6932
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 335187ea6932
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 542c537ae837
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 542c537ae837
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 468189d3f4fd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 468189d3f4fd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 64d376138548
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 64d376138548
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ece88dd198cb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ece88dd198cb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fbc07a90f45a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fbc07a90f45a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9dacc8411efc
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9dacc8411efc
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf5601e06cea
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf5601e06cea
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b078c66906e7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b078c66906e7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6811bd114b43
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6811bd114b43
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 83ec1d774aed
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 83ec1d774aed
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 417712bd3432
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 417712bd3432
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f722d7abd2c1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f722d7abd2c1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ffe374ee0187
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ffe374ee0187
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 72d5feac0789
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 72d5feac0789
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3f5ce8c0c758
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3f5ce8c0c758
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c0dff7f229cb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c0dff7f229cb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d56b2d8feb0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d56b2d8feb0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 019a3c726af2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 019a3c726af2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9960074411e3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9960074411e3
2024-03-10 19:28:08 proc_lsst.multi:146 [INFO]  found job 16982500 in provider escience_a40
proc_lsst.multi INFO: found job 16982500 in provider escience_a40
2024-03-10 19:28:08 proc_lsst.multi:201 [INFO]  cancelling 16982500 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982500 on provider escience_a40
2024-03-10 19:28:08 proc_lsst.multi:146 [INFO]  found job 16982499 in provider escience_a40
proc_lsst.multi INFO: found job 16982499 in provider escience_a40
2024-03-10 19:28:08 proc_lsst.multi:201 [INFO]  cancelling 16982499 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982499 on provider escience_a40
2024-03-10 19:28:08 proc_lsst.multi:146 [INFO]  found job 16982501 in provider escience_a40
proc_lsst.multi INFO: found job 16982501 in provider escience_a40
2024-03-10 19:28:08 proc_lsst.multi:201 [INFO]  cancelling 16982501 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982501 on provider escience_a40
2024-03-10 19:28:08 proc_lsst.multi:146 [INFO]  found job 16982502 in provider escience_a40
proc_lsst.multi INFO: found job 16982502 in provider escience_a40
2024-03-10 19:28:08 proc_lsst.multi:201 [INFO]  cancelling 16982502 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982502 on provider escience_a40
2024-03-10 19:28:08 proc_lsst.multi:146 [INFO]  found job 16982474 in provider astro
proc_lsst.multi INFO: found job 16982474 in provider astro
2024-03-10 19:28:08 proc_lsst.multi:201 [INFO]  cancelling 16982474 on provider astro
proc_lsst.multi INFO: cancelling 16982474 on provider astro
2024-03-10 19:28:08 proc_lsst.multi:146 [INFO]  found job 16982480 in provider astro
proc_lsst.multi INFO: found job 16982480 in provider astro
2024-03-10 19:28:08 proc_lsst.multi:201 [INFO]  cancelling 16982480 on provider astro
proc_lsst.multi INFO: cancelling 16982480 on provider astro
2024-03-10 19:28:09 proc_lsst.multi:146 [INFO]  found job 16982456 in provider astro
proc_lsst.multi INFO: found job 16982456 in provider astro
2024-03-10 19:28:09 proc_lsst.multi:201 [INFO]  cancelling 16982456 on provider astro
proc_lsst.multi INFO: cancelling 16982456 on provider astro
2024-03-10 19:28:09 proc_lsst.multi:146 [INFO]  found job 16982472 in provider astro
proc_lsst.multi INFO: found job 16982472 in provider astro
2024-03-10 19:28:09 proc_lsst.multi:201 [INFO]  cancelling 16982472 on provider astro
proc_lsst.multi INFO: cancelling 16982472 on provider astro
2024-03-10 19:28:09 proc_lsst.multi:146 [INFO]  found job 16982468 in provider astro
proc_lsst.multi INFO: found job 16982468 in provider astro
2024-03-10 19:28:09 proc_lsst.multi:201 [INFO]  cancelling 16982468 on provider astro
proc_lsst.multi INFO: cancelling 16982468 on provider astro
2024-03-10 19:28:09 proc_lsst.multi:146 [INFO]  found job 16982466 in provider astro
proc_lsst.multi INFO: found job 16982466 in provider astro
2024-03-10 19:28:09 proc_lsst.multi:201 [INFO]  cancelling 16982466 on provider astro
proc_lsst.multi INFO: cancelling 16982466 on provider astro
2024-03-10 19:28:09 proc_lsst.multi:146 [INFO]  found job 16982457 in provider astro
proc_lsst.multi INFO: found job 16982457 in provider astro
2024-03-10 19:28:09 proc_lsst.multi:201 [INFO]  cancelling 16982457 on provider astro
proc_lsst.multi INFO: cancelling 16982457 on provider astro
2024-03-10 19:28:09 proc_lsst.multi:146 [INFO]  found job 16982496 in provider escience_a40
proc_lsst.multi INFO: found job 16982496 in provider escience_a40
2024-03-10 19:28:09 proc_lsst.multi:201 [INFO]  cancelling 16982496 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982496 on provider escience_a40
2024-03-10 19:28:10 proc_lsst.multi:146 [INFO]  found job 16982459 in provider astro
proc_lsst.multi INFO: found job 16982459 in provider astro
2024-03-10 19:28:10 proc_lsst.multi:201 [INFO]  cancelling 16982459 on provider astro
proc_lsst.multi INFO: cancelling 16982459 on provider astro
2024-03-10 19:28:10 proc_lsst.multi:146 [INFO]  found job 16982461 in provider astro
proc_lsst.multi INFO: found job 16982461 in provider astro
2024-03-10 19:28:10 proc_lsst.multi:201 [INFO]  cancelling 16982461 on provider astro
proc_lsst.multi INFO: cancelling 16982461 on provider astro
2024-03-10 19:28:10 proc_lsst.multi:146 [INFO]  found job 16982458 in provider astro
proc_lsst.multi INFO: found job 16982458 in provider astro
2024-03-10 19:28:10 proc_lsst.multi:201 [INFO]  cancelling 16982458 on provider astro
proc_lsst.multi INFO: cancelling 16982458 on provider astro
2024-03-10 19:28:10 proc_lsst.multi:146 [INFO]  found job 16982494 in provider escience_a40
proc_lsst.multi INFO: found job 16982494 in provider escience_a40
2024-03-10 19:28:10 proc_lsst.multi:201 [INFO]  cancelling 16982494 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982494 on provider escience_a40
2024-03-10 19:28:10 proc_lsst.multi:146 [INFO]  found job 16982497 in provider escience_a40
proc_lsst.multi INFO: found job 16982497 in provider escience_a40
2024-03-10 19:28:10 proc_lsst.multi:201 [INFO]  cancelling 16982497 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982497 on provider escience_a40
2024-03-10 19:28:10 proc_lsst.multi:146 [INFO]  found job 16982498 in provider escience_a40
proc_lsst.multi INFO: found job 16982498 in provider escience_a40
2024-03-10 19:28:10 proc_lsst.multi:201 [INFO]  cancelling 16982498 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982498 on provider escience_a40
2024-03-10 19:28:11 proc_lsst.multi:146 [INFO]  found job 16982493 in provider escience_a40
proc_lsst.multi INFO: found job 16982493 in provider escience_a40
2024-03-10 19:28:11 proc_lsst.multi:201 [INFO]  cancelling 16982493 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982493 on provider escience_a40
2024-03-10 19:28:11 proc_lsst.multi:146 [INFO]  found job 16982486 in provider astro
proc_lsst.multi INFO: found job 16982486 in provider astro
2024-03-10 19:28:11 proc_lsst.multi:201 [INFO]  cancelling 16982486 on provider astro
proc_lsst.multi INFO: cancelling 16982486 on provider astro
2024-03-10 19:28:11 proc_lsst.multi:146 [INFO]  found job 16982495 in provider escience_a40
proc_lsst.multi INFO: found job 16982495 in provider escience_a40
2024-03-10 19:28:11 proc_lsst.multi:201 [INFO]  cancelling 16982495 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982495 on provider escience_a40
2024-03-10 19:28:11 proc_lsst.multi:146 [INFO]  found job 16982492 in provider escience_a40
proc_lsst.multi INFO: found job 16982492 in provider escience_a40
2024-03-10 19:28:11 proc_lsst.multi:201 [INFO]  cancelling 16982492 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982492 on provider escience_a40
2024-03-10 19:28:11 proc_lsst.multi:146 [INFO]  found job 16982463 in provider astro
proc_lsst.multi INFO: found job 16982463 in provider astro
2024-03-10 19:28:11 proc_lsst.multi:201 [INFO]  cancelling 16982463 on provider astro
proc_lsst.multi INFO: cancelling 16982463 on provider astro
2024-03-10 19:28:11 proc_lsst.multi:146 [INFO]  found job 16982478 in provider astro
proc_lsst.multi INFO: found job 16982478 in provider astro
2024-03-10 19:28:11 proc_lsst.multi:201 [INFO]  cancelling 16982478 on provider astro
proc_lsst.multi INFO: cancelling 16982478 on provider astro
2024-03-10 19:28:12 proc_lsst.multi:146 [INFO]  found job 16982471 in provider astro
proc_lsst.multi INFO: found job 16982471 in provider astro
2024-03-10 19:28:12 proc_lsst.multi:201 [INFO]  cancelling 16982471 on provider astro
proc_lsst.multi INFO: cancelling 16982471 on provider astro
2024-03-10 19:28:12 proc_lsst.multi:146 [INFO]  found job 16982483 in provider astro
proc_lsst.multi INFO: found job 16982483 in provider astro
2024-03-10 19:28:12 proc_lsst.multi:201 [INFO]  cancelling 16982483 on provider astro
proc_lsst.multi INFO: cancelling 16982483 on provider astro
2024-03-10 19:28:12 proc_lsst.multi:146 [INFO]  found job 16982485 in provider astro
proc_lsst.multi INFO: found job 16982485 in provider astro
2024-03-10 19:28:12 proc_lsst.multi:201 [INFO]  cancelling 16982485 on provider astro
proc_lsst.multi INFO: cancelling 16982485 on provider astro
2024-03-10 19:28:12 proc_lsst.multi:146 [INFO]  found job 16982481 in provider astro
proc_lsst.multi INFO: found job 16982481 in provider astro
2024-03-10 19:28:12 proc_lsst.multi:201 [INFO]  cancelling 16982481 on provider astro
proc_lsst.multi INFO: cancelling 16982481 on provider astro
2024-03-10 19:28:12 proc_lsst.multi:146 [INFO]  found job 16982482 in provider astro
proc_lsst.multi INFO: found job 16982482 in provider astro
2024-03-10 19:28:12 proc_lsst.multi:201 [INFO]  cancelling 16982482 on provider astro
proc_lsst.multi INFO: cancelling 16982482 on provider astro
2024-03-10 19:28:12 proc_lsst.multi:146 [INFO]  found job 16982484 in provider astro
proc_lsst.multi INFO: found job 16982484 in provider astro
2024-03-10 19:28:12 proc_lsst.multi:201 [INFO]  cancelling 16982484 on provider astro
proc_lsst.multi INFO: cancelling 16982484 on provider astro
2024-03-10 19:28:13 proc_lsst.multi:146 [INFO]  found job 96597 in provider local
proc_lsst.multi INFO: found job 96597 in provider local
2024-03-10 19:28:13 proc_lsst.multi:201 [INFO]  cancelling 96597 on provider local
proc_lsst.multi INFO: cancelling 96597 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 96597
2024-03-10 19:28:13 proc_lsst.multi:146 [INFO]  found job 16982479 in provider astro
proc_lsst.multi INFO: found job 16982479 in provider astro
2024-03-10 19:28:13 proc_lsst.multi:201 [INFO]  cancelling 16982479 on provider astro
proc_lsst.multi INFO: cancelling 16982479 on provider astro
2024-03-10 19:28:13 proc_lsst.multi:146 [INFO]  found job 16982470 in provider astro
proc_lsst.multi INFO: found job 16982470 in provider astro
2024-03-10 19:28:13 proc_lsst.multi:201 [INFO]  cancelling 16982470 on provider astro
proc_lsst.multi INFO: cancelling 16982470 on provider astro
2024-03-10 19:28:13 proc_lsst.multi:146 [INFO]  found job 16982491 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982491 in provider escience_rtx6k
2024-03-10 19:28:13 proc_lsst.multi:201 [INFO]  cancelling 16982491 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982491 on provider escience_rtx6k
2024-03-10 19:28:13 proc_lsst.multi:146 [INFO]  found job 16982469 in provider astro
proc_lsst.multi INFO: found job 16982469 in provider astro
2024-03-10 19:28:13 proc_lsst.multi:201 [INFO]  cancelling 16982469 on provider astro
proc_lsst.multi INFO: cancelling 16982469 on provider astro
2024-03-10 19:28:13 proc_lsst.multi:146 [INFO]  found job 16982489 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982489 in provider escience_rtx6k
2024-03-10 19:28:13 proc_lsst.multi:201 [INFO]  cancelling 16982489 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982489 on provider escience_rtx6k
2024-03-10 19:28:14 proc_lsst.multi:146 [INFO]  found job 16982467 in provider astro
proc_lsst.multi INFO: found job 16982467 in provider astro
2024-03-10 19:28:14 proc_lsst.multi:201 [INFO]  cancelling 16982467 on provider astro
proc_lsst.multi INFO: cancelling 16982467 on provider astro
2024-03-10 19:28:14 proc_lsst.multi:146 [INFO]  found job 16982488 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982488 in provider escience_rtx6k
2024-03-10 19:28:14 proc_lsst.multi:201 [INFO]  cancelling 16982488 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982488 on provider escience_rtx6k
2024-03-10 19:28:14 proc_lsst.multi:146 [INFO]  found job 16982490 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982490 in provider escience_rtx6k
2024-03-10 19:28:14 proc_lsst.multi:201 [INFO]  cancelling 16982490 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982490 on provider escience_rtx6k
2024-03-10 19:28:14 proc_lsst.multi:146 [INFO]  found job 16982476 in provider astro
proc_lsst.multi INFO: found job 16982476 in provider astro
2024-03-10 19:28:14 proc_lsst.multi:201 [INFO]  cancelling 16982476 on provider astro
proc_lsst.multi INFO: cancelling 16982476 on provider astro
2024-03-10 19:28:14 proc_lsst.multi:146 [INFO]  found job 16982475 in provider astro
proc_lsst.multi INFO: found job 16982475 in provider astro
2024-03-10 19:28:14 proc_lsst.multi:201 [INFO]  cancelling 16982475 on provider astro
proc_lsst.multi INFO: cancelling 16982475 on provider astro
2024-03-10 19:28:14 proc_lsst.multi:146 [INFO]  found job 16982477 in provider astro
proc_lsst.multi INFO: found job 16982477 in provider astro
2024-03-10 19:28:14 proc_lsst.multi:201 [INFO]  cancelling 16982477 on provider astro
proc_lsst.multi INFO: cancelling 16982477 on provider astro
2024-03-10 19:28:15 proc_lsst.multi:146 [INFO]  found job 16982465 in provider astro
proc_lsst.multi INFO: found job 16982465 in provider astro
2024-03-10 19:28:15 proc_lsst.multi:201 [INFO]  cancelling 16982465 on provider astro
proc_lsst.multi INFO: cancelling 16982465 on provider astro
2024-03-10 19:28:15 proc_lsst.multi:146 [INFO]  found job 16982462 in provider astro
proc_lsst.multi INFO: found job 16982462 in provider astro
2024-03-10 19:28:15 proc_lsst.multi:201 [INFO]  cancelling 16982462 on provider astro
proc_lsst.multi INFO: cancelling 16982462 on provider astro
2024-03-10 19:28:15 proc_lsst.multi:146 [INFO]  found job 16982464 in provider astro
proc_lsst.multi INFO: found job 16982464 in provider astro
2024-03-10 19:28:15 proc_lsst.multi:201 [INFO]  cancelling 16982464 on provider astro
proc_lsst.multi INFO: cancelling 16982464 on provider astro
2024-03-10 19:28:17 proc_lsst.multi:146 [INFO]  found job 16982487 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982487 in provider escience_rtx6k
2024-03-10 19:28:17 proc_lsst.multi:201 [INFO]  cancelling 16982487 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982487 on provider escience_rtx6k
2024-03-10 19:28:17 proc_lsst.multi:146 [INFO]  found job 16982473 in provider astro
proc_lsst.multi INFO: found job 16982473 in provider astro
2024-03-10 19:28:17 proc_lsst.multi:201 [INFO]  cancelling 16982473 on provider astro
proc_lsst.multi INFO: cancelling 16982473 on provider astro
parsl.dataflow.dflow INFO: Shutting down executor multi
2024-03-10 19:28:17 proc_lsst.multi:40 [INFO]  Cancelling all provider resources
proc_lsst.multi INFO: Cancelling all provider resources
2024-03-10 19:28:17 proc_lsst.multi:47 [INFO]  new jobs since last cancel ['16982456', '16982457', '16982458', '16982459', '16982461', '16982462', '16982463', '16982464', '16982465', '16982466', '16982467', '16982468', '16982469', '16982470', '16982471', '16982472', '16982473', '16982474', '16982475', '16982476', '16982477', '16982478', '16982479', '16982480', '16982481', '16982482', '16982483', '16982484', '16982485', '16982486', '16982492', '16982493', '16982494', '16982495', '16982496', '16982497', '16982498', '16982499', '16982500', '16982501', '16982502', '16982487', '16982488', '16982489', '16982490', '16982491', '96597']
proc_lsst.multi INFO: new jobs since last cancel ['16982456', '16982457', '16982458', '16982459', '16982461', '16982462', '16982463', '16982464', '16982465', '16982466', '16982467', '16982468', '16982469', '16982470', '16982471', '16982472', '16982473', '16982474', '16982475', '16982476', '16982477', '16982478', '16982479', '16982480', '16982481', '16982482', '16982483', '16982484', '16982485', '16982486', '16982492', '16982493', '16982494', '16982495', '16982496', '16982497', '16982498', '16982499', '16982500', '16982501', '16982502', '16982487', '16982488', '16982489', '16982490', '16982491', '96597']
2024-03-10 19:28:17 proc_lsst.multi:146 [INFO]  found job 16982456 in provider astro
proc_lsst.multi INFO: found job 16982456 in provider astro
2024-03-10 19:28:17 proc_lsst.multi:201 [INFO]  cancelling 16982456 on provider astro
proc_lsst.multi INFO: cancelling 16982456 on provider astro
2024-03-10 19:28:17 proc_lsst.multi:146 [INFO]  found job 16982457 in provider astro
proc_lsst.multi INFO: found job 16982457 in provider astro
2024-03-10 19:28:17 proc_lsst.multi:201 [INFO]  cancelling 16982457 on provider astro
proc_lsst.multi INFO: cancelling 16982457 on provider astro
2024-03-10 19:28:18 proc_lsst.multi:146 [INFO]  found job 16982458 in provider astro
proc_lsst.multi INFO: found job 16982458 in provider astro
2024-03-10 19:28:18 proc_lsst.multi:201 [INFO]  cancelling 16982458 on provider astro
proc_lsst.multi INFO: cancelling 16982458 on provider astro
2024-03-10 19:28:18 proc_lsst.multi:146 [INFO]  found job 16982459 in provider astro
proc_lsst.multi INFO: found job 16982459 in provider astro
2024-03-10 19:28:18 proc_lsst.multi:201 [INFO]  cancelling 16982459 on provider astro
proc_lsst.multi INFO: cancelling 16982459 on provider astro
2024-03-10 19:28:18 proc_lsst.multi:146 [INFO]  found job 16982461 in provider astro
proc_lsst.multi INFO: found job 16982461 in provider astro
2024-03-10 19:28:18 proc_lsst.multi:201 [INFO]  cancelling 16982461 on provider astro
proc_lsst.multi INFO: cancelling 16982461 on provider astro
2024-03-10 19:28:18 proc_lsst.multi:146 [INFO]  found job 16982462 in provider astro
proc_lsst.multi INFO: found job 16982462 in provider astro
2024-03-10 19:28:18 proc_lsst.multi:201 [INFO]  cancelling 16982462 on provider astro
proc_lsst.multi INFO: cancelling 16982462 on provider astro
2024-03-10 19:28:18 proc_lsst.multi:146 [INFO]  found job 16982463 in provider astro
proc_lsst.multi INFO: found job 16982463 in provider astro
2024-03-10 19:28:18 proc_lsst.multi:201 [INFO]  cancelling 16982463 on provider astro
proc_lsst.multi INFO: cancelling 16982463 on provider astro
2024-03-10 19:28:18 proc_lsst.multi:146 [INFO]  found job 16982464 in provider astro
proc_lsst.multi INFO: found job 16982464 in provider astro
2024-03-10 19:28:18 proc_lsst.multi:201 [INFO]  cancelling 16982464 on provider astro
proc_lsst.multi INFO: cancelling 16982464 on provider astro
2024-03-10 19:28:19 proc_lsst.multi:146 [INFO]  found job 16982465 in provider astro
proc_lsst.multi INFO: found job 16982465 in provider astro
2024-03-10 19:28:19 proc_lsst.multi:201 [INFO]  cancelling 16982465 on provider astro
proc_lsst.multi INFO: cancelling 16982465 on provider astro
2024-03-10 19:28:19 proc_lsst.multi:146 [INFO]  found job 16982466 in provider astro
proc_lsst.multi INFO: found job 16982466 in provider astro
2024-03-10 19:28:19 proc_lsst.multi:201 [INFO]  cancelling 16982466 on provider astro
proc_lsst.multi INFO: cancelling 16982466 on provider astro
2024-03-10 19:28:19 proc_lsst.multi:146 [INFO]  found job 16982467 in provider astro
proc_lsst.multi INFO: found job 16982467 in provider astro
2024-03-10 19:28:19 proc_lsst.multi:201 [INFO]  cancelling 16982467 on provider astro
proc_lsst.multi INFO: cancelling 16982467 on provider astro
2024-03-10 19:28:19 proc_lsst.multi:146 [INFO]  found job 16982468 in provider astro
proc_lsst.multi INFO: found job 16982468 in provider astro
2024-03-10 19:28:19 proc_lsst.multi:201 [INFO]  cancelling 16982468 on provider astro
proc_lsst.multi INFO: cancelling 16982468 on provider astro
2024-03-10 19:28:19 proc_lsst.multi:146 [INFO]  found job 16982469 in provider astro
proc_lsst.multi INFO: found job 16982469 in provider astro
2024-03-10 19:28:19 proc_lsst.multi:201 [INFO]  cancelling 16982469 on provider astro
proc_lsst.multi INFO: cancelling 16982469 on provider astro
2024-03-10 19:28:19 proc_lsst.multi:146 [INFO]  found job 16982470 in provider astro
proc_lsst.multi INFO: found job 16982470 in provider astro
2024-03-10 19:28:19 proc_lsst.multi:201 [INFO]  cancelling 16982470 on provider astro
proc_lsst.multi INFO: cancelling 16982470 on provider astro
2024-03-10 19:28:20 proc_lsst.multi:146 [INFO]  found job 16982471 in provider astro
proc_lsst.multi INFO: found job 16982471 in provider astro
2024-03-10 19:28:20 proc_lsst.multi:201 [INFO]  cancelling 16982471 on provider astro
proc_lsst.multi INFO: cancelling 16982471 on provider astro
2024-03-10 19:28:20 proc_lsst.multi:146 [INFO]  found job 16982472 in provider astro
proc_lsst.multi INFO: found job 16982472 in provider astro
2024-03-10 19:28:20 proc_lsst.multi:201 [INFO]  cancelling 16982472 on provider astro
proc_lsst.multi INFO: cancelling 16982472 on provider astro
2024-03-10 19:28:20 proc_lsst.multi:146 [INFO]  found job 16982473 in provider astro
proc_lsst.multi INFO: found job 16982473 in provider astro
2024-03-10 19:28:20 proc_lsst.multi:201 [INFO]  cancelling 16982473 on provider astro
proc_lsst.multi INFO: cancelling 16982473 on provider astro
2024-03-10 19:28:20 proc_lsst.multi:146 [INFO]  found job 16982474 in provider astro
proc_lsst.multi INFO: found job 16982474 in provider astro
2024-03-10 19:28:20 proc_lsst.multi:201 [INFO]  cancelling 16982474 on provider astro
proc_lsst.multi INFO: cancelling 16982474 on provider astro
2024-03-10 19:28:20 proc_lsst.multi:146 [INFO]  found job 16982475 in provider astro
proc_lsst.multi INFO: found job 16982475 in provider astro
2024-03-10 19:28:20 proc_lsst.multi:201 [INFO]  cancelling 16982475 on provider astro
proc_lsst.multi INFO: cancelling 16982475 on provider astro
2024-03-10 19:28:21 proc_lsst.multi:146 [INFO]  found job 16982476 in provider astro
proc_lsst.multi INFO: found job 16982476 in provider astro
2024-03-10 19:28:21 proc_lsst.multi:201 [INFO]  cancelling 16982476 on provider astro
proc_lsst.multi INFO: cancelling 16982476 on provider astro
2024-03-10 19:28:21 proc_lsst.multi:146 [INFO]  found job 16982477 in provider astro
proc_lsst.multi INFO: found job 16982477 in provider astro
2024-03-10 19:28:21 proc_lsst.multi:201 [INFO]  cancelling 16982477 on provider astro
proc_lsst.multi INFO: cancelling 16982477 on provider astro
2024-03-10 19:28:21 proc_lsst.multi:146 [INFO]  found job 16982478 in provider astro
proc_lsst.multi INFO: found job 16982478 in provider astro
2024-03-10 19:28:21 proc_lsst.multi:201 [INFO]  cancelling 16982478 on provider astro
proc_lsst.multi INFO: cancelling 16982478 on provider astro
2024-03-10 19:28:21 proc_lsst.multi:146 [INFO]  found job 16982479 in provider astro
proc_lsst.multi INFO: found job 16982479 in provider astro
2024-03-10 19:28:21 proc_lsst.multi:201 [INFO]  cancelling 16982479 on provider astro
proc_lsst.multi INFO: cancelling 16982479 on provider astro
2024-03-10 19:28:21 proc_lsst.multi:146 [INFO]  found job 16982480 in provider astro
proc_lsst.multi INFO: found job 16982480 in provider astro
2024-03-10 19:28:21 proc_lsst.multi:201 [INFO]  cancelling 16982480 on provider astro
proc_lsst.multi INFO: cancelling 16982480 on provider astro
2024-03-10 19:28:21 proc_lsst.multi:146 [INFO]  found job 16982481 in provider astro
proc_lsst.multi INFO: found job 16982481 in provider astro
2024-03-10 19:28:21 proc_lsst.multi:201 [INFO]  cancelling 16982481 on provider astro
proc_lsst.multi INFO: cancelling 16982481 on provider astro
2024-03-10 19:28:22 proc_lsst.multi:146 [INFO]  found job 16982482 in provider astro
proc_lsst.multi INFO: found job 16982482 in provider astro
2024-03-10 19:28:22 proc_lsst.multi:201 [INFO]  cancelling 16982482 on provider astro
proc_lsst.multi INFO: cancelling 16982482 on provider astro
2024-03-10 19:28:22 proc_lsst.multi:146 [INFO]  found job 16982483 in provider astro
proc_lsst.multi INFO: found job 16982483 in provider astro
2024-03-10 19:28:22 proc_lsst.multi:201 [INFO]  cancelling 16982483 on provider astro
proc_lsst.multi INFO: cancelling 16982483 on provider astro
2024-03-10 19:28:22 proc_lsst.multi:146 [INFO]  found job 16982484 in provider astro
proc_lsst.multi INFO: found job 16982484 in provider astro
2024-03-10 19:28:22 proc_lsst.multi:201 [INFO]  cancelling 16982484 on provider astro
proc_lsst.multi INFO: cancelling 16982484 on provider astro
2024-03-10 19:28:22 proc_lsst.multi:146 [INFO]  found job 16982485 in provider astro
proc_lsst.multi INFO: found job 16982485 in provider astro
2024-03-10 19:28:22 proc_lsst.multi:201 [INFO]  cancelling 16982485 on provider astro
proc_lsst.multi INFO: cancelling 16982485 on provider astro
2024-03-10 19:28:22 proc_lsst.multi:146 [INFO]  found job 16982486 in provider astro
proc_lsst.multi INFO: found job 16982486 in provider astro
2024-03-10 19:28:22 proc_lsst.multi:201 [INFO]  cancelling 16982486 on provider astro
proc_lsst.multi INFO: cancelling 16982486 on provider astro
2024-03-10 19:28:22 proc_lsst.multi:146 [INFO]  found job 16982492 in provider escience_a40
proc_lsst.multi INFO: found job 16982492 in provider escience_a40
2024-03-10 19:28:22 proc_lsst.multi:201 [INFO]  cancelling 16982492 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982492 on provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:146 [INFO]  found job 16982493 in provider escience_a40
proc_lsst.multi INFO: found job 16982493 in provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:201 [INFO]  cancelling 16982493 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982493 on provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:146 [INFO]  found job 16982494 in provider escience_a40
proc_lsst.multi INFO: found job 16982494 in provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:201 [INFO]  cancelling 16982494 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982494 on provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:146 [INFO]  found job 16982495 in provider escience_a40
proc_lsst.multi INFO: found job 16982495 in provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:201 [INFO]  cancelling 16982495 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982495 on provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:146 [INFO]  found job 16982496 in provider escience_a40
proc_lsst.multi INFO: found job 16982496 in provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:201 [INFO]  cancelling 16982496 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982496 on provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:146 [INFO]  found job 16982497 in provider escience_a40
proc_lsst.multi INFO: found job 16982497 in provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:201 [INFO]  cancelling 16982497 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982497 on provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:146 [INFO]  found job 16982498 in provider escience_a40
proc_lsst.multi INFO: found job 16982498 in provider escience_a40
2024-03-10 19:28:23 proc_lsst.multi:201 [INFO]  cancelling 16982498 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982498 on provider escience_a40
2024-03-10 19:28:24 proc_lsst.multi:146 [INFO]  found job 16982499 in provider escience_a40
proc_lsst.multi INFO: found job 16982499 in provider escience_a40
2024-03-10 19:28:24 proc_lsst.multi:201 [INFO]  cancelling 16982499 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982499 on provider escience_a40
2024-03-10 19:28:24 proc_lsst.multi:146 [INFO]  found job 16982500 in provider escience_a40
proc_lsst.multi INFO: found job 16982500 in provider escience_a40
2024-03-10 19:28:24 proc_lsst.multi:201 [INFO]  cancelling 16982500 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982500 on provider escience_a40
2024-03-10 19:28:24 proc_lsst.multi:146 [INFO]  found job 16982501 in provider escience_a40
proc_lsst.multi INFO: found job 16982501 in provider escience_a40
2024-03-10 19:28:24 proc_lsst.multi:201 [INFO]  cancelling 16982501 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982501 on provider escience_a40
2024-03-10 19:28:24 proc_lsst.multi:146 [INFO]  found job 16982502 in provider escience_a40
proc_lsst.multi INFO: found job 16982502 in provider escience_a40
2024-03-10 19:28:24 proc_lsst.multi:201 [INFO]  cancelling 16982502 on provider escience_a40
proc_lsst.multi INFO: cancelling 16982502 on provider escience_a40
2024-03-10 19:28:24 proc_lsst.multi:146 [INFO]  found job 16982487 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982487 in provider escience_rtx6k
2024-03-10 19:28:24 proc_lsst.multi:201 [INFO]  cancelling 16982487 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982487 on provider escience_rtx6k
2024-03-10 19:28:24 proc_lsst.multi:146 [INFO]  found job 16982488 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982488 in provider escience_rtx6k
2024-03-10 19:28:24 proc_lsst.multi:201 [INFO]  cancelling 16982488 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982488 on provider escience_rtx6k
2024-03-10 19:28:25 proc_lsst.multi:146 [INFO]  found job 16982489 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982489 in provider escience_rtx6k
2024-03-10 19:28:25 proc_lsst.multi:201 [INFO]  cancelling 16982489 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982489 on provider escience_rtx6k
2024-03-10 19:28:25 proc_lsst.multi:146 [INFO]  found job 16982490 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982490 in provider escience_rtx6k
2024-03-10 19:28:25 proc_lsst.multi:201 [INFO]  cancelling 16982490 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982490 on provider escience_rtx6k
2024-03-10 19:28:25 proc_lsst.multi:146 [INFO]  found job 16982491 in provider escience_rtx6k
proc_lsst.multi INFO: found job 16982491 in provider escience_rtx6k
2024-03-10 19:28:25 proc_lsst.multi:201 [INFO]  cancelling 16982491 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 16982491 on provider escience_rtx6k
2024-03-10 19:28:25 proc_lsst.multi:146 [INFO]  found job 96597 in provider local
proc_lsst.multi INFO: found job 96597 in provider local
2024-03-10 19:28:25 proc_lsst.multi:201 [INFO]  cancelling 96597 on provider local
proc_lsst.multi INFO: cancelling 96597 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 96597
parsl.providers.local.local WARNING: Failed to kill PID: 96597 and child processes on local
2024-03-10 19:28:26 proc_lsst.multi:50 [INFO]  no new jobs since last cancel, resuming executor shutdown
proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown
parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown
parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt
parsl.dataflow.dflow INFO: Shut down executor multi
parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal
parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete
parsl.executors.threads DEBUG: Done with executor shutdown
parsl.dataflow.dflow INFO: Shut down executor _parsl_internal
parsl.dataflow.dflow INFO: Terminated executors
parsl.dataflow.dflow INFO: DFK cleanup complete
parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread
lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 240.7512 seconds
lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190707_B0b_science#step0_20240310T192404Z' submitted for execution with id 'None'
lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 240.7631 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.002 Gibyte
lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 257.6839 seconds; current memory usage: 0.272 Gibyte, delta: 0.135 Gibyte, peak delta: 0.137 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.273 Gibyte (largest child process)
Run Id: None
Run Name: DEEP_20190707_B0b_science#step0_20240310T192404Z
parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up

real	4m20.437s
user	0m55.964s
sys	0m26.013s