+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit
+ bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20201018/B1b --output-run DEEP/20201018/B1b/science#step0/20240314T025028Z --qgraph pipeline.qgraph
lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process.
lsst.ctrl.bps.drivers INFO: Starting submission process
lsst.ctrl.bps.drivers INFO: Initializing execution environment
lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 8.9456 seconds; current memory usage: 0.178 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.191 Gibyte (main), 0.000 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph)
lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.5224 seconds
lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.5397 seconds
lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/pipeline_orig.qgraph DEEP/20201018/B1b/science#step0/20240314T025028Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/pipeline.qgraph 
lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.4837 seconds
lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 8.0598 seconds; current memory usage: 0.270 Gibyte, delta: 0.092 Gibyte, peak delta: 0.080 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.262 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs)
lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0099 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.262 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 186 cluster(s)
lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow)
lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20201018_B1b_science#step0_20240314T025028Z'
lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.5216 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.262 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 188 job(s) (including final)
lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow)
parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses
    s_addresses.add(address_by_interface(interface))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface
    return socket.inet_ntoa(fcntl.ioctl(
                            ^^^^^^^^^^^^
OSError: [Errno 99] Cannot assign requested address
lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z
lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 5.7459 seconds; current memory usage: 0.272 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.262 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting submit stage
lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution
parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000
parsl.dataflow.dflow INFO: Starting DataFlowKernel with config
Config(
    app_cache=True, 
    checkpoint_files=None, 
    checkpoint_mode='task_exit', 
    checkpoint_period=None, 
    executors=[MultiHighThroughputExecutor()], 
    garbage_collect=True, 
    initialize_logging=True, 
    internal_tasks_max_threads=10, 
    max_idletime=120.0, 
    monitoring=None, 
    retries=1, 
    retry_handler=None, 
    run_dir='runinfo', 
    strategy='simple', 
    usage_tracking=False
)
parsl.dataflow.dflow INFO: Parsl version: 2023.06.12
parsl.usage_tracking.usage DEBUG: Tracking status: False
parsl.dataflow.dflow INFO: Run id is: a41e6d22-18c8-4a67-b752-5d3768d4802c
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py
parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name
parsl.dataflow.memoization INFO: App caching initialized
parsl.dataflow.strategy DEBUG: Scaling strategy: simple
parsl.executors.high_throughput.executor DEBUG: Starting queue management thread
parsl.executors.high_throughput.executor DEBUG: queue management worker starting
parsl.executors.high_throughput.executor DEBUG: Started queue management thread
Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.executors.high_throughput.executor DEBUG: Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 23253589051136)>
parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider:
<proc_lsst.multi.MultiProvider object at 0x1526273b9ad0>
parsl.executors.status_handling INFO: Scaling out by 1 blocks
parsl.executors.status_handling INFO: Allocated block ID 0
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0
2024-03-14 02:51:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
2024-03-14 02:51:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-14 02:51:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
parsl.providers.local.local DEBUG: Launching in remote mode
2024-03-14 02:51:15 proc_lsst.multi:170 [INFO]  [multi] job_id 13859
proc_lsst.multi INFO: [multi] job_id 13859
2024-03-14 02:51:15 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:15 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:15 proc_lsst.multi:178 [INFO]  [multi] provider local accepted submit and returned 13859
proc_lsst.multi INFO: [multi] provider local accepted submit and returned 13859
parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 13859
parsl.dataflow.job_status_poller DEBUG: Adding executor multi
parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: <AppFuture at 0x1526273469d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 86fae69a7ffe77c8150cfd565c07c623
parsl.dataflow.memoization INFO: Task 0 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/178acbf3-feef-467d-a18f-55a4173b7e0f_isrForCrosstalkSources_946686_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/178acbf3-feef-467d-a18f-55a4173b7e0f_isrForCrosstalkSources_946686_30.stderr
parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: <AppFuture at 0x15262806f0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 0270d8e774767babec74314ae0ddb81e
parsl.dataflow.memoization INFO: Task 1 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/a224889e-50f9-461b-9ac6-201084ef6afe_isrForCrosstalkSources_946687_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/a224889e-50f9-461b-9ac6-201084ef6afe_isrForCrosstalkSources_946687_45.stderr
parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: <AppFuture at 0x152627335a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 2 has memoization hash d1c9300894989449c1cc0ee318c32c11
parsl.dataflow.memoization INFO: Task 2 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ddd6f3f8-8249-4ca2-aca0-ad090a0a1cbb_isrForCrosstalkSources_946687_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ddd6f3f8-8249-4ca2-aca0-ad090a0a1cbb_isrForCrosstalkSources_946687_34.stderr
parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: <AppFuture at 0x1526318ee150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 3b46af84bfa8ab435cfca231e676bef4
parsl.dataflow.memoization INFO: Task 3 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b03e98d8-bd45-4ebd-8328-57c283312bf2_isrForCrosstalkSources_946688_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b03e98d8-bd45-4ebd-8328-57c283312bf2_isrForCrosstalkSources_946688_47.stderr
parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: <AppFuture at 0x152627358590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 4 has memoization hash b81ff6b798f72eba4828b2e4b8ca1298
parsl.dataflow.memoization INFO: Task 4 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/3c30339b-870a-40c7-bf30-df3275bcbbbf_isrForCrosstalkSources_946688_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/3c30339b-870a-40c7-bf30-df3275bcbbbf_isrForCrosstalkSources_946688_22.stderr
parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: <AppFuture at 0x15262734c310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 8862012b117e9e90bd01604ab8cda678
parsl.dataflow.memoization INFO: Task 5 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f6272109-9a1a-4404-8865-90e4baa663b1_isrForCrosstalkSources_946686_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f6272109-9a1a-4404-8865-90e4baa663b1_isrForCrosstalkSources_946686_1.stderr
parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: <AppFuture at 0x1526273bb8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 3e0147e7732c885e2d65530cbc20a154
parsl.dataflow.memoization INFO: Task 6 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/39fc9517-fd22-4130-884f-e0611838c68b_isrForCrosstalkSources_946688_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/39fc9517-fd22-4130-884f-e0611838c68b_isrForCrosstalkSources_946688_7.stderr
parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: <AppFuture at 0x152627322f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 7 has memoization hash dc540036bd9df469063d82989d75d309
parsl.dataflow.memoization INFO: Task 7 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/9f5dbf9a-10ad-4a2b-b4d4-e682ae7f4a99_isrForCrosstalkSources_946687_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/9f5dbf9a-10ad-4a2b-b4d4-e682ae7f4a99_isrForCrosstalkSources_946687_9.stderr
parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: <AppFuture at 0x152627320610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 4ef30cb1a1286ab70643b256b6abc745
parsl.dataflow.memoization INFO: Task 8 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cf5bf996-cf93-4a21-bcaf-7b7e8f202cdf_isrForCrosstalkSources_946686_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cf5bf996-cf93-4a21-bcaf-7b7e8f202cdf_isrForCrosstalkSources_946686_20.stderr
parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: <AppFuture at 0x152627320410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 70e8580c231f5aaa67b91675c92e7646
parsl.dataflow.memoization INFO: Task 9 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/8eb91ebc-eeac-42d8-bb32-ad68682b853a_isrForCrosstalkSources_946686_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/8eb91ebc-eeac-42d8-bb32-ad68682b853a_isrForCrosstalkSources_946686_54.stderr
parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: <AppFuture at 0x152627321b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 10 has memoization hash c4d9f0d200c1e73c7f8257dd312facdd
parsl.dataflow.memoization INFO: Task 10 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b7e75e57-710d-4e1f-9ee7-22502b76ecfb_isrForCrosstalkSources_946688_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b7e75e57-710d-4e1f-9ee7-22502b76ecfb_isrForCrosstalkSources_946688_2.stderr
parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: <AppFuture at 0x15262731a990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 19072fe979f6d1aead4196cb6402d699
parsl.dataflow.memoization INFO: Task 11 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x15262711fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d71f7cbe-7f4a-4419-a93e-78b197d7d723_isrForCrosstalkSources_946686_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d71f7cbe-7f4a-4419-a93e-78b197d7d723_isrForCrosstalkSources_946686_48.stderr
parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: <AppFuture at 0x15262731b250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 6f7396e907299f27d4425b46ab1c3f8d
parsl.dataflow.memoization INFO: Task 12 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e7b1e40b-4fc3-4f38-9630-e673fef8d429_isrForCrosstalkSources_946688_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e7b1e40b-4fc3-4f38-9630-e673fef8d429_isrForCrosstalkSources_946688_27.stderr
parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: <AppFuture at 0x152627319850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 13 has memoization hash e7a9d0011f83abfe6fbc2ea233245b5b
parsl.dataflow.memoization INFO: Task 13 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271604a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/926e74a7-012d-498a-8404-8dfe6ae5ebaf_isrForCrosstalkSources_946688_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/926e74a7-012d-498a-8404-8dfe6ae5ebaf_isrForCrosstalkSources_946688_36.stderr
parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: <AppFuture at 0x15262757d090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 62b57d39b7d04cbc291a8465196997d6
parsl.dataflow.memoization INFO: Task 14 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/7046e8a0-04e9-4e56-a7b3-9a7f2f7d3b5c_isrForCrosstalkSources_946687_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/7046e8a0-04e9-4e56-a7b3-9a7f2f7d3b5c_isrForCrosstalkSources_946687_27.stderr
parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: <AppFuture at 0x15262730e250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 183cbd48bb669b10e3b001b199a433a5
parsl.dataflow.memoization INFO: Task 15 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/319648de-ef82-46fb-bf8d-f0f47c8cde45_isrForCrosstalkSources_946686_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/319648de-ef82-46fb-bf8d-f0f47c8cde45_isrForCrosstalkSources_946686_38.stderr
parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: <AppFuture at 0x15262730e810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 6b9c15fbdb23daa132468a8348ae578f
parsl.dataflow.memoization INFO: Task 16 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271605e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/22494117-4be5-40ef-98d9-8d3da433eeac_isrForCrosstalkSources_946688_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/22494117-4be5-40ef-98d9-8d3da433eeac_isrForCrosstalkSources_946688_3.stderr
parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: <AppFuture at 0x15262730c9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 2e4e1353c0b686029b5776047e31ad28
parsl.dataflow.memoization INFO: Task 17 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/3d10df6f-f4b2-42f3-a84e-dfbb8ac49a87_isrForCrosstalkSources_946688_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/3d10df6f-f4b2-42f3-a84e-dfbb8ac49a87_isrForCrosstalkSources_946688_10.stderr
parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: <AppFuture at 0x15262730e010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 18 has memoization hash c542096f043a58edf936e5e4732739ff
parsl.dataflow.memoization INFO: Task 18 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271607c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/d3c991f1-b6c5-4a01-be1b-512435fc2ff2_isrForCrosstalkSources_946687_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/d3c991f1-b6c5-4a01-be1b-512435fc2ff2_isrForCrosstalkSources_946687_15.stderr
parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: <AppFuture at 0x152627307d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 2e46ba575311041c0b6616209771840c
parsl.dataflow.memoization INFO: Task 19 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/177153dc-0c37-4719-beb1-c75330f8d763_isrForCrosstalkSources_946687_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/177153dc-0c37-4719-beb1-c75330f8d763_isrForCrosstalkSources_946687_26.stderr
parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: <AppFuture at 0x152627306f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 6cf9aed0eea8e3a7ee277513bc7c09a1
parsl.dataflow.memoization INFO: Task 20 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fd29e63f-5a91-4207-b028-7eba491e6dbb_isrForCrosstalkSources_946687_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fd29e63f-5a91-4207-b028-7eba491e6dbb_isrForCrosstalkSources_946687_37.stderr
parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: <AppFuture at 0x152627306390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 208742476e792804d310701dafd11d44
parsl.dataflow.memoization INFO: Task 21 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ca37e732-57c2-4920-9180-614f4a12fb08_isrForCrosstalkSources_946686_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ca37e732-57c2-4920-9180-614f4a12fb08_isrForCrosstalkSources_946686_17.stderr
parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: <AppFuture at 0x1526272fffd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 2f319d5a039bca0eab7803b96e5dc278
parsl.dataflow.memoization INFO: Task 22 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271609a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ba4ca7ff-615e-4a7b-9c9a-bc74dd235835_isrForCrosstalkSources_946687_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ba4ca7ff-615e-4a7b-9c9a-bc74dd235835_isrForCrosstalkSources_946687_61.stderr
parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: <AppFuture at 0x1526272fe010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 43013db986104468288d30d138202702
parsl.dataflow.memoization INFO: Task 23 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/dfa3be0f-23fe-48aa-8925-5130ff0774b5_isrForCrosstalkSources_946687_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/dfa3be0f-23fe-48aa-8925-5130ff0774b5_isrForCrosstalkSources_946687_53.stderr
parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: <AppFuture at 0x1526272fc250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 24 has memoization hash e1b21fdec2516dc6da4f029196dc708f
parsl.dataflow.memoization INFO: Task 24 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/beebdcec-3409-449e-bf04-933d5114b31c_isrForCrosstalkSources_946687_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/beebdcec-3409-449e-bf04-933d5114b31c_isrForCrosstalkSources_946687_60.stderr
parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: <AppFuture at 0x1526272fded0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 25 has memoization hash e83371ed0884d5e602226c3cbea9cf24
parsl.dataflow.memoization INFO: Task 25 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/47ce4e8a-ed92-45bb-a03c-9ed357fa345a_isrForCrosstalkSources_946687_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/47ce4e8a-ed92-45bb-a03c-9ed357fa345a_isrForCrosstalkSources_946687_3.stderr
parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: <AppFuture at 0x1526272f4450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 01e23e84ec20b8900d3ec3427d2677d7
parsl.dataflow.memoization INFO: Task 26 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/19adc1b4-8230-46b3-bf95-adf328c99aa8_isrForCrosstalkSources_946687_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/19adc1b4-8230-46b3-bf95-adf328c99aa8_isrForCrosstalkSources_946687_40.stderr
parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: <AppFuture at 0x1526272f4250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 850b7e02262e39c8cc97f58275d76751
parsl.dataflow.memoization INFO: Task 27 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/f5eb4652-71e0-4ebb-9744-df7ef63c4234_isrForCrosstalkSources_946688_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/f5eb4652-71e0-4ebb-9744-df7ef63c4234_isrForCrosstalkSources_946688_12.stderr
parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: <AppFuture at 0x1526272f4990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 28 has memoization hash fa8c60b99dcd40cc3c6771e9476c6e31
parsl.dataflow.memoization INFO: Task 28 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/186a47a4-85d4-4837-aa65-631a1cf44891_isrForCrosstalkSources_946687_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/186a47a4-85d4-4837-aa65-631a1cf44891_isrForCrosstalkSources_946687_54.stderr
parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: <AppFuture at 0x1526272f4c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 24b5d41bb52e2f3736076e6c3581097c
parsl.dataflow.memoization INFO: Task 29 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/b52aff84-6573-4348-b68b-ccb5386e83e2_isrForCrosstalkSources_946687_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/b52aff84-6573-4348-b68b-ccb5386e83e2_isrForCrosstalkSources_946687_19.stderr
parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: <AppFuture at 0x1526272e9450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 30 has memoization hash cf85724ed16a0e01a09f47ea690d134f
parsl.dataflow.memoization INFO: Task 30 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/48f5a42a-9fe1-4aa5-838a-9fcdb4a2c91a_isrForCrosstalkSources_946688_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/48f5a42a-9fe1-4aa5-838a-9fcdb4a2c91a_isrForCrosstalkSources_946688_13.stderr
parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: <AppFuture at 0x1526272ea450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 8fc32393f8b7047ff0acd0e2961b7570
parsl.dataflow.memoization INFO: Task 31 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8af63715-54d1-4c8e-b73e-dd4a07f147c8_isrForCrosstalkSources_946688_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8af63715-54d1-4c8e-b73e-dd4a07f147c8_isrForCrosstalkSources_946688_24.stderr
parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: <AppFuture at 0x1526272e9410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 3d8af0adec26d1de835a7158e029831e
parsl.dataflow.memoization INFO: Task 32 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e8a1154a-04c6-4238-acc0-e14d18f88661_isrForCrosstalkSources_946687_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e8a1154a-04c6-4238-acc0-e14d18f88661_isrForCrosstalkSources_946687_30.stderr
parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: <AppFuture at 0x1526274ca590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 33 has memoization hash b86581f6f97416baf4664d64d7b6488a
parsl.dataflow.memoization INFO: Task 33 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/bfad661f-c0df-4b3c-a796-6d128d40bc4c_isrForCrosstalkSources_946688_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/bfad661f-c0df-4b3c-a796-6d128d40bc4c_isrForCrosstalkSources_946688_8.stderr
parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: <AppFuture at 0x1526274cb090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 34 has memoization hash c2bd29eaa1e54e9da0b9867a260236ec
parsl.dataflow.memoization INFO: Task 34 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271611c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/5b60a9f9-e31d-45e1-913a-21e9b83d5740_isrForCrosstalkSources_946687_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/5b60a9f9-e31d-45e1-913a-21e9b83d5740_isrForCrosstalkSources_946687_11.stderr
parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: <AppFuture at 0x1526274ca110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 843b1524209fe2f2671ec390cdd9a403
parsl.dataflow.memoization INFO: Task 35 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cd70a8c6-c39b-483a-a7a6-30610879565c_isrForCrosstalkSources_946686_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cd70a8c6-c39b-483a-a7a6-30610879565c_isrForCrosstalkSources_946686_60.stderr
parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: <AppFuture at 0x1526274cbc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 62a0ae01823f2b87da7f00f4044be5aa
parsl.dataflow.memoization INFO: Task 36 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271613a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/247ad116-a7b6-4e54-9215-038b9d62f3b8_isrForCrosstalkSources_946686_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/247ad116-a7b6-4e54-9215-038b9d62f3b8_isrForCrosstalkSources_946686_22.stderr
parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: <AppFuture at 0x1526274ca790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 37 has memoization hash ab1081c46f41247c72e4428f1472dc1b
parsl.dataflow.memoization INFO: Task 37 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f4286ed7-f4d9-4244-93fb-c9413ce9b205_isrForCrosstalkSources_946687_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f4286ed7-f4d9-4244-93fb-c9413ce9b205_isrForCrosstalkSources_946687_51.stderr
parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: <AppFuture at 0x1526274cee90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 83dd2c2666c590043b963745c728064c
parsl.dataflow.memoization INFO: Task 38 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5aa21ed4-1bc6-46ef-a416-a59145d0a8d4_isrForCrosstalkSources_946688_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5aa21ed4-1bc6-46ef-a416-a59145d0a8d4_isrForCrosstalkSources_946688_48.stderr
parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: <AppFuture at 0x1526274cc890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 88eefd379f7d21e8b82b32fe3c993621
parsl.dataflow.memoization INFO: Task 39 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271614e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/3f4b60f6-7913-45ce-87b3-ab1ba543d9e2_isrForCrosstalkSources_946686_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/3f4b60f6-7913-45ce-87b3-ab1ba543d9e2_isrForCrosstalkSources_946686_34.stderr
parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: <AppFuture at 0x1526272df790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 407a2b61a6f800209b6b9133fdf19a17
parsl.dataflow.memoization INFO: Task 40 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/88f86847-c867-4882-bd71-159dd2fdc431_isrForCrosstalkSources_946687_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/88f86847-c867-4882-bd71-159dd2fdc431_isrForCrosstalkSources_946687_42.stderr
parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: <AppFuture at 0x1526272df490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 37e2ca372ff09923b408635d4bd27bfd
parsl.dataflow.memoization INFO: Task 41 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271616c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/4cf75b3f-57f7-4916-961b-5deaa43327a2_isrForCrosstalkSources_946688_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/4cf75b3f-57f7-4916-961b-5deaa43327a2_isrForCrosstalkSources_946688_44.stderr
parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: <AppFuture at 0x1526272dd750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 7afef2ca224e2da7d325c1973e130c39
parsl.dataflow.memoization INFO: Task 42 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/291e3683-916f-4009-aa27-fcf12cb0ae0f_isrForCrosstalkSources_946686_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/291e3683-916f-4009-aa27-fcf12cb0ae0f_isrForCrosstalkSources_946686_57.stderr
parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: <AppFuture at 0x1526274c9610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 43 has memoization hash d1777a423f119ceced9c667942a77f03
parsl.dataflow.memoization INFO: Task 43 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2059dee1-3638-4fc7-b3d6-679b2650d02a_isrForCrosstalkSources_946688_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2059dee1-3638-4fc7-b3d6-679b2650d02a_isrForCrosstalkSources_946688_1.stderr
parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: <AppFuture at 0x152631982d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 163a64f2d21196c285ad883e32f284ba
parsl.dataflow.memoization INFO: Task 44 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/90f67fcb-d50d-47ed-b0ab-ff165b6492e5_isrForCrosstalkSources_946688_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/90f67fcb-d50d-47ed-b0ab-ff165b6492e5_isrForCrosstalkSources_946688_61.stderr
parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: <AppFuture at 0x152631981710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 69c15957b5be9f3f14cb47124f74ae73
parsl.dataflow.memoization INFO: Task 45 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271618a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/bffca539-9f82-49d4-96d5-5b38232bb1a3_isrForCrosstalkSources_946688_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/bffca539-9f82-49d4-96d5-5b38232bb1a3_isrForCrosstalkSources_946688_11.stderr
parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: <AppFuture at 0x152631983c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 7a80b16887b4d0ef603aa8a78b99b013
parsl.dataflow.memoization INFO: Task 46 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271619e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/d65d5469-3472-4a5d-be0a-e493f5f2815b_isrForCrosstalkSources_946688_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/d65d5469-3472-4a5d-be0a-e493f5f2815b_isrForCrosstalkSources_946688_46.stderr
parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: <AppFuture at 0x1526319825d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 435811480da25987a9b0c34491a28cd7
parsl.dataflow.memoization INFO: Task 47 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/55be47dc-55e4-404c-836e-61e2973f7c08_isrForCrosstalkSources_946686_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/55be47dc-55e4-404c-836e-61e2973f7c08_isrForCrosstalkSources_946686_18.stderr
parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: <AppFuture at 0x15262725edd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 4cecce01e7e46a224556d5180c2b18b5
parsl.dataflow.memoization INFO: Task 48 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2c5611b7-52e7-4fd7-9d12-7f6680a677bb_isrForCrosstalkSources_946688_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2c5611b7-52e7-4fd7-9d12-7f6680a677bb_isrForCrosstalkSources_946688_54.stderr
parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: <AppFuture at 0x15262725ced0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 49 has memoization hash c6b8f083a51bb5769cf4556e1cbe660f
parsl.dataflow.memoization INFO: Task 49 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/39b24d0c-2282-4d90-b71e-effeb9815fef_isrForCrosstalkSources_946688_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/39b24d0c-2282-4d90-b71e-effeb9815fef_isrForCrosstalkSources_946688_60.stderr
parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: <AppFuture at 0x152627100250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 50 has memoization hash b023b2ddd737d1ce5a213e751dc7a8f4
parsl.dataflow.memoization INFO: Task 50 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/6cf5f3a0-8d61-4d77-a12c-324ac74942ab_isrForCrosstalkSources_946687_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/6cf5f3a0-8d61-4d77-a12c-324ac74942ab_isrForCrosstalkSources_946687_56.stderr
parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: <AppFuture at 0x152627103590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 51 has memoization hash f873a87a26e8a5a99580d026e7a14d8e
parsl.dataflow.memoization INFO: Task 51 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d9396dd8-3ce9-4eaa-b5a2-6b8db69348fb_isrForCrosstalkSources_946686_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d9396dd8-3ce9-4eaa-b5a2-6b8db69348fb_isrForCrosstalkSources_946686_39.stderr
parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: <AppFuture at 0x1526271003d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 4eafc9010df716e84afe9116516beafa
parsl.dataflow.memoization INFO: Task 52 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/740dd3d5-b9e5-4c60-9b09-e6c5d862ef67_isrForCrosstalkSources_946687_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/740dd3d5-b9e5-4c60-9b09-e6c5d862ef67_isrForCrosstalkSources_946687_52.stderr
parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: <AppFuture at 0x152627319b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 86cd7b28de2cc6f9523da881efe19ad3
parsl.dataflow.memoization INFO: Task 53 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/57eb1bc3-ddef-4ba9-87bc-b07c5db00b2e_isrForCrosstalkSources_946686_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/57eb1bc3-ddef-4ba9-87bc-b07c5db00b2e_isrForCrosstalkSources_946686_45.stderr
parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: <AppFuture at 0x152627112b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 54 has memoization hash f66680b9d8425f47e9e6f34f9fc5ef93
parsl.dataflow.memoization INFO: Task 54 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/14b273ed-8049-4d84-aaf4-d839ab050467_isrForCrosstalkSources_946686_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/14b273ed-8049-4d84-aaf4-d839ab050467_isrForCrosstalkSources_946686_23.stderr
parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: <AppFuture at 0x152627110cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 6a22024297b38593ced66f71b92407e8
parsl.dataflow.memoization INFO: Task 55 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/02d9ac39-19b7-44f7-982e-f167a5dfc4de_isrForCrosstalkSources_946687_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/02d9ac39-19b7-44f7-982e-f167a5dfc4de_isrForCrosstalkSources_946687_55.stderr
parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: <AppFuture at 0x152627111510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 02603638b1d3ddf38cab5c4a9bb49f0a
parsl.dataflow.memoization INFO: Task 56 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f39e930a-a837-45df-b8c8-a6a419c58962_isrForCrosstalkSources_946687_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f39e930a-a837-45df-b8c8-a6a419c58962_isrForCrosstalkSources_946687_35.stderr
parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: <AppFuture at 0x152627113a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 0b4126b6ae7efbf0a56254fc858838a1
parsl.dataflow.memoization INFO: Task 57 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627161f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/8e88f2b6-07ae-4bc5-89e4-098d89ea164e_isrForCrosstalkSources_946687_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/8e88f2b6-07ae-4bc5-89e4-098d89ea164e_isrForCrosstalkSources_946687_25.stderr
parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: <AppFuture at 0x152627113850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 36480f15708c6dfe308fedcd3d454fa4
parsl.dataflow.memoization INFO: Task 58 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271620c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/bbc705c2-da61-47fc-8b7d-8cb998f3e08c_isrForCrosstalkSources_946686_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/bbc705c2-da61-47fc-8b7d-8cb998f3e08c_isrForCrosstalkSources_946686_9.stderr
parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: <AppFuture at 0x152627194d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 3ab559416f61c59be25a4abd9f44bac7
parsl.dataflow.memoization INFO: Task 59 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/7e3ff471-f68b-4659-ac98-1cae70a0ebb3_isrForCrosstalkSources_946688_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/7e3ff471-f68b-4659-ac98-1cae70a0ebb3_isrForCrosstalkSources_946688_41.stderr
parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: <AppFuture at 0x152627194850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 0fa26d2e9808b6d971bbe0228d0ff291
parsl.dataflow.memoization INFO: Task 60 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e4d628af-dab3-41b9-b451-d27dfa8f629b_isrForCrosstalkSources_946686_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e4d628af-dab3-41b9-b451-d27dfa8f629b_isrForCrosstalkSources_946686_51.stderr
parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: <AppFuture at 0x1526271956d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 3b77ea52b2cb2c144a20668876ac9dca
parsl.dataflow.memoization INFO: Task 61 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/56bb7a21-eb14-4eab-b776-2349adde6b2f_isrForCrosstalkSources_946687_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/56bb7a21-eb14-4eab-b776-2349adde6b2f_isrForCrosstalkSources_946687_41.stderr
parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: <AppFuture at 0x152627195b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 62 has memoization hash aefe7fadf828f39ebfa696575758c0ff
parsl.dataflow.memoization INFO: Task 62 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271622a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4d1617dc-4a0f-4418-9562-f27d06b15fcb_isrForCrosstalkSources_946687_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4d1617dc-4a0f-4418-9562-f27d06b15fcb_isrForCrosstalkSources_946687_23.stderr
parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: <AppFuture at 0x152627196810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 63 has memoization hash ba53bc12a745a2dbfbde9a7c16a5c0ba
parsl.dataflow.memoization INFO: Task 63 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271623e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/50f54a4a-708f-4ec3-bccf-8508f748ce6d_isrForCrosstalkSources_946686_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/50f54a4a-708f-4ec3-bccf-8508f748ce6d_isrForCrosstalkSources_946686_58.stderr
parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: <AppFuture at 0x152627197310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 6b31d1fec6d2e804c3b2da0abe2ba119
parsl.dataflow.memoization INFO: Task 64 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2d3c9e1d-81a1-415f-9960-2f2be0412ba0_isrForCrosstalkSources_946688_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2d3c9e1d-81a1-415f-9960-2f2be0412ba0_isrForCrosstalkSources_946688_15.stderr
parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: <AppFuture at 0x152627197890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 7047f84bc93be00db117e8b33cf1d207
parsl.dataflow.memoization INFO: Task 65 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/7d4d37a3-3631-448e-886f-be689d212c16_isrForCrosstalkSources_946686_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/7d4d37a3-3631-448e-886f-be689d212c16_isrForCrosstalkSources_946686_28.stderr
parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: <AppFuture at 0x1526272dfd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 66 has memoization hash d526bd0eed523260fb761371351d01a5
parsl.dataflow.memoization INFO: Task 66 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2957f4e7-f6d1-4a05-8c7f-615713fcbe74_isrForCrosstalkSources_946688_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2957f4e7-f6d1-4a05-8c7f-615713fcbe74_isrForCrosstalkSources_946688_25.stderr
parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: <AppFuture at 0x15262719cc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 608a050d1ed836f8128fa0fdc303bb8f
parsl.dataflow.memoization INFO: Task 67 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271625c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/23f2166a-52fb-4458-b731-4f5ed588fbdd_isrForCrosstalkSources_946687_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/23f2166a-52fb-4458-b731-4f5ed588fbdd_isrForCrosstalkSources_946687_17.stderr
parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: <AppFuture at 0x15262719d4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 68 has memoization hash ab1b3766390edaece7784d25d86cf338
parsl.dataflow.memoization INFO: Task 68 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f55311a3-52d5-41e0-ac2d-43b2f5b40f68_isrForCrosstalkSources_946686_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f55311a3-52d5-41e0-ac2d-43b2f5b40f68_isrForCrosstalkSources_946686_27.stderr
parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: <AppFuture at 0x15262719e150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 6d9ab095c2f396844ac98fb622ce2da3
parsl.dataflow.memoization INFO: Task 69 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271627a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/476177f6-3b8c-4b6d-bbf2-9e51fdd5972f_isrForCrosstalkSources_946687_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/476177f6-3b8c-4b6d-bbf2-9e51fdd5972f_isrForCrosstalkSources_946687_62.stderr
parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: <AppFuture at 0x15262719e890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 1a2c5ab2535fc46bcac297eca67aaeec
parsl.dataflow.memoization INFO: Task 70 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/203a81ef-1bc0-4225-9f99-9aef2301577a_isrForCrosstalkSources_946688_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/203a81ef-1bc0-4225-9f99-9aef2301577a_isrForCrosstalkSources_946688_31.stderr
parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: <AppFuture at 0x15262719e850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 62259e181d6407c2ac36024c41946cf8
parsl.dataflow.memoization INFO: Task 71 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/3ca88fdf-15cf-416c-8dc4-a5b807653343_isrForCrosstalkSources_946687_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/3ca88fdf-15cf-416c-8dc4-a5b807653343_isrForCrosstalkSources_946687_5.stderr
parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: <AppFuture at 0x15262719f650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 62e060279a86e1bd7c019f30b8f54989
parsl.dataflow.memoization INFO: Task 72 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271628e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/7a3663cd-5514-4196-a47b-714d003007e3_isrForCrosstalkSources_946688_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/7a3663cd-5514-4196-a47b-714d003007e3_isrForCrosstalkSources_946688_58.stderr
parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: <AppFuture at 0x152627110610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 0e4cc3a9c68da3e1a26201bd9ce83a8c
parsl.dataflow.memoization INFO: Task 73 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f18c156f-b97a-44ba-b91c-b28554badaa6_isrForCrosstalkSources_946686_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f18c156f-b97a-44ba-b91c-b28554badaa6_isrForCrosstalkSources_946686_12.stderr
parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: <AppFuture at 0x15262719cb10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 1a1cf5f8a489e00f88a81ad89a10da3e
parsl.dataflow.memoization INFO: Task 74 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/60361637-f405-478b-bdc1-e231901a3644_isrForCrosstalkSources_946688_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/60361637-f405-478b-bdc1-e231901a3644_isrForCrosstalkSources_946688_19.stderr
parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: <AppFuture at 0x152627101590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 8ece0ee3159cb6612b959ef23f68a187
parsl.dataflow.memoization INFO: Task 75 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/12923e8b-424b-4ba0-8648-5abacd9549ae_isrForCrosstalkSources_946686_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/12923e8b-424b-4ba0-8648-5abacd9549ae_isrForCrosstalkSources_946686_26.stderr
parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: <AppFuture at 0x1526271a9e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 76 has memoization hash f5a22a9c56e32d0986ee23dabd799b70
parsl.dataflow.memoization INFO: Task 76 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8f827929-5de1-43d6-ade4-73b8985b1de1_isrForCrosstalkSources_946688_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8f827929-5de1-43d6-ade4-73b8985b1de1_isrForCrosstalkSources_946688_39.stderr
parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: <AppFuture at 0x1526271aa310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 77 has memoization hash d8305f7e2e9031d16724858f11164914
parsl.dataflow.memoization INFO: Task 77 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/1899f4b0-d615-4eff-af8e-182f005c52bb_isrForCrosstalkSources_946687_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/1899f4b0-d615-4eff-af8e-182f005c52bb_isrForCrosstalkSources_946687_50.stderr
parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: <AppFuture at 0x1526271aba10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 91172527ee9a218deed63cbc3fe72594
parsl.dataflow.memoization INFO: Task 78 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/77290212-df59-40a4-8337-d2084f8d0981_isrForCrosstalkSources_946686_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/77290212-df59-40a4-8337-d2084f8d0981_isrForCrosstalkSources_946686_42.stderr
parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: <AppFuture at 0x1526271aaa50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 0613623d760411f3162d311afe868c8a
parsl.dataflow.memoization INFO: Task 79 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c9fb1fa7-bc54-4bc5-83cc-e8e0f33ea670_isrForCrosstalkSources_946688_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c9fb1fa7-bc54-4bc5-83cc-e8e0f33ea670_isrForCrosstalkSources_946688_32.stderr
parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: <AppFuture at 0x1526271abe50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 80 has memoization hash c8bbbe6a768c52132687eb0b1541032a
parsl.dataflow.memoization INFO: Task 80 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/24a1ab8d-5f6b-4337-8f30-53a649fd550a_isrForCrosstalkSources_946686_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/24a1ab8d-5f6b-4337-8f30-53a649fd550a_isrForCrosstalkSources_946686_53.stderr
parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: <AppFuture at 0x1526271bc450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 81 has memoization hash c1b28d195b29168594a8d8d4b2dd2e9d
parsl.dataflow.memoization INFO: Task 81 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/45861880-6309-45d8-82e7-ca2890a513fc_isrForCrosstalkSources_946688_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/45861880-6309-45d8-82e7-ca2890a513fc_isrForCrosstalkSources_946688_5.stderr
parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: <AppFuture at 0x1526271bcd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 82 has memoization hash ea984c5f0065a537d75db287dbb9dd23
parsl.dataflow.memoization INFO: Task 82 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627162f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/1cb3f4d5-441d-43a1-b8a7-3ba214b6e9c4_isrForCrosstalkSources_946686_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/1cb3f4d5-441d-43a1-b8a7-3ba214b6e9c4_isrForCrosstalkSources_946686_59.stderr
parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: <AppFuture at 0x1526271be710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 5796357cf552a45506af5f69cc8b26da
parsl.dataflow.memoization INFO: Task 83 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4632149f-5b29-4530-a42b-772a3e1bcf46_isrForCrosstalkSources_946686_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4632149f-5b29-4530-a42b-772a3e1bcf46_isrForCrosstalkSources_946686_4.stderr
parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: <AppFuture at 0x1526271be090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 84 has memoization hash dd6a283af6ab13c28e8ab75a69a1e95c
parsl.dataflow.memoization INFO: Task 84 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/df0c8c75-25f6-4466-92e1-5082d62fd173_isrForCrosstalkSources_946688_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/df0c8c75-25f6-4466-92e1-5082d62fd173_isrForCrosstalkSources_946688_14.stderr
parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: <AppFuture at 0x1526271bed90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 76d797a4a7f94673b778f254edf8c7dd
parsl.dataflow.memoization INFO: Task 85 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271631a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/623742d7-02bb-4907-bcdc-107e94956c8b_isrForCrosstalkSources_946687_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/623742d7-02bb-4907-bcdc-107e94956c8b_isrForCrosstalkSources_946687_12.stderr
parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: <AppFuture at 0x1526271bf090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 52b86a30d7f4bef9c7dd2c9a9f451ed4
parsl.dataflow.memoization INFO: Task 86 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271632e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/5db29991-e888-426d-9116-5f116c845b60_isrForCrosstalkSources_946687_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/5db29991-e888-426d-9116-5f116c845b60_isrForCrosstalkSources_946687_28.stderr
parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: <AppFuture at 0x1526271be390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 7f830b8784850ca158498c138bfa3feb
parsl.dataflow.memoization INFO: Task 87 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f897d711-d7b3-4acd-9ff7-8b0d1336e43a_isrForCrosstalkSources_946687_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f897d711-d7b3-4acd-9ff7-8b0d1336e43a_isrForCrosstalkSources_946687_2.stderr
parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: <AppFuture at 0x1526271c0b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 5bc93e9307f26c27e393e4107389f79b
parsl.dataflow.memoization INFO: Task 88 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/909239b8-a4fb-43b7-9656-6e0083b8210c_isrForCrosstalkSources_946687_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/909239b8-a4fb-43b7-9656-6e0083b8210c_isrForCrosstalkSources_946687_31.stderr
parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: <AppFuture at 0x1526271c1710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 89 has memoization hash c8578be6dcd3ff73cf97ccfe3104640a
parsl.dataflow.memoization INFO: Task 89 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/1a9e623a-0349-4aa8-a343-5ffcc7121ba2_isrForCrosstalkSources_946686_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/1a9e623a-0349-4aa8-a343-5ffcc7121ba2_isrForCrosstalkSources_946686_14.stderr
parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: <AppFuture at 0x1526271c1110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 35b475516d2c1e0476196208db0cc1a5
parsl.dataflow.memoization INFO: Task 90 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271634c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e2fa4a26-bf27-480c-a09a-72d14a1a2e52_isrForCrosstalkSources_946687_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e2fa4a26-bf27-480c-a09a-72d14a1a2e52_isrForCrosstalkSources_946687_7.stderr
parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: <AppFuture at 0x1526271c1bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 4586d918ebcc25a8da3096828ae390fb
parsl.dataflow.memoization INFO: Task 91 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/12a605c1-f712-4e99-8235-39e00dd3bc22_isrForCrosstalkSources_946688_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/12a605c1-f712-4e99-8235-39e00dd3bc22_isrForCrosstalkSources_946688_16.stderr
parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: <AppFuture at 0x1526271c3a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 4b661b7e4ca115996f7f588f7f7f5ef0
parsl.dataflow.memoization INFO: Task 92 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ed036f12-b8f5-4d0a-9bc3-2ede82268269_isrForCrosstalkSources_946686_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ed036f12-b8f5-4d0a-9bc3-2ede82268269_isrForCrosstalkSources_946686_40.stderr
parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: <AppFuture at 0x1526271c36d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 9c227fa17606aaeffadb6f230efdb58e
parsl.dataflow.memoization INFO: Task 93 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271636a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/3a639762-854b-4fc1-9858-9d17f237d0e0_isrForCrosstalkSources_946687_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/3a639762-854b-4fc1-9858-9d17f237d0e0_isrForCrosstalkSources_946687_4.stderr
parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: <AppFuture at 0x1526271c3ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 9d0e0fa1c673b5f8203bbe375d7f7a60
parsl.dataflow.memoization INFO: Task 94 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/6d343fd2-347f-4471-937a-c9d5579be555_isrForCrosstalkSources_946688_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/6d343fd2-347f-4471-937a-c9d5579be555_isrForCrosstalkSources_946688_59.stderr
parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: <AppFuture at 0x152627194410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 04bd669faa15694914e89d250da431e1
parsl.dataflow.memoization INFO: Task 95 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271637e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e14804c3-817d-475a-ad38-b2541d15f9f1_isrForCrosstalkSources_946688_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e14804c3-817d-475a-ad38-b2541d15f9f1_isrForCrosstalkSources_946688_62.stderr
parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: <AppFuture at 0x1526271cce90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 96 has memoization hash d88347f351310b97f1d7783a8caec828
parsl.dataflow.memoization INFO: Task 96 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/498d9138-8790-494a-b340-fbebe1a31499_isrForCrosstalkSources_946686_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/498d9138-8790-494a-b340-fbebe1a31499_isrForCrosstalkSources_946686_5.stderr
parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: <AppFuture at 0x1526271cda50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 97 has memoization hash f9b3aa243f2d0c3035fc4eac94c24485
parsl.dataflow.memoization INFO: Task 97 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/5714a688-6733-407e-b2d2-b43056b14af9_isrForCrosstalkSources_946686_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/5714a688-6733-407e-b2d2-b43056b14af9_isrForCrosstalkSources_946686_21.stderr
parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: <AppFuture at 0x1526271cd850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 813938d29b40f4514e309b32b8bbc032
parsl.dataflow.memoization INFO: Task 98 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x1526271639c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/9eb6fcc8-3468-4c7e-982c-b5798d9776eb_isrForCrosstalkSources_946688_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/9eb6fcc8-3468-4c7e-982c-b5798d9776eb_isrForCrosstalkSources_946688_55.stderr
parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: <AppFuture at 0x1526271ce910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 8a18aae9d33be85090628738faab021e
parsl.dataflow.memoization INFO: Task 99 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/38d88613-3dec-4a7b-ba83-6ee823ed4523_isrForCrosstalkSources_946686_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/38d88613-3dec-4a7b-ba83-6ee823ed4523_isrForCrosstalkSources_946686_31.stderr
parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: <AppFuture at 0x1526271cf450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 8d254dbf4881fb11be0e817249213dae
parsl.dataflow.memoization INFO: Task 100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e5f1bc64-6320-4a06-b9fe-60057b164ea1_isrForCrosstalkSources_946687_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e5f1bc64-6320-4a06-b9fe-60057b164ea1_isrForCrosstalkSources_946687_1.stderr
parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: <AppFuture at 0x1526271cfc50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 101 has memoization hash ea105c1b42722569de53ba4ec0634aa3
parsl.dataflow.memoization INFO: Task 101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/035fca6d-148a-4ffc-87a6-ca3f7323d72c_isrForCrosstalkSources_946687_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/035fca6d-148a-4ffc-87a6-ca3f7323d72c_isrForCrosstalkSources_946687_36.stderr
parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: <AppFuture at 0x152625ba48d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 102 has memoization hash a8af4f037570f497155da1a553ada4c5
parsl.dataflow.memoization INFO: Task 102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8ab96b33-6a33-40c1-80fc-08943489bee7_isrForCrosstalkSources_946688_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8ab96b33-6a33-40c1-80fc-08943489bee7_isrForCrosstalkSources_946688_20.stderr
parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: <AppFuture at 0x152625ba4ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 74e2954c3770b2aea6e362c314903e14
parsl.dataflow.memoization INFO: Task 103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/300bd0c3-f1d7-4c4d-a30f-c8786d78e2a0_isrForCrosstalkSources_946687_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/300bd0c3-f1d7-4c4d-a30f-c8786d78e2a0_isrForCrosstalkSources_946687_47.stderr
parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: <AppFuture at 0x152625ba56d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 2363189b3fd96902761ee09ac7f086fa
parsl.dataflow.memoization INFO: Task 104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/879da37e-67f4-4f9a-b00b-a51192798824_isrForCrosstalkSources_946688_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/879da37e-67f4-4f9a-b00b-a51192798824_isrForCrosstalkSources_946688_18.stderr
parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: <AppFuture at 0x152625ba5ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 043d5a3681155a16eb71e91684f0f508
parsl.dataflow.memoization INFO: Task 105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/6cdf7f25-053b-4ff7-825a-6d6ccf4c9b4c_isrForCrosstalkSources_946686_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/6cdf7f25-053b-4ff7-825a-6d6ccf4c9b4c_isrForCrosstalkSources_946686_62.stderr
parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: <AppFuture at 0x152625ba6990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 106 has memoization hash f85756f1b8dd849a61fba1e96caac77b
parsl.dataflow.memoization INFO: Task 106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627163f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/acf0863a-f03e-48f5-92d0-a3da4797f14d_isrForCrosstalkSources_946686_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/acf0863a-f03e-48f5-92d0-a3da4797f14d_isrForCrosstalkSources_946686_6.stderr
parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: <AppFuture at 0x152625ba7bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 107 has memoization hash ca1f67aee2fe8b4fb7077f2063d8ceb2
parsl.dataflow.memoization INFO: Task 107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152627160720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/6be3df15-136f-47ab-8222-c6269bd4b3b4_isrForCrosstalkSources_946688_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/6be3df15-136f-47ab-8222-c6269bd4b3b4_isrForCrosstalkSources_946688_26.stderr
parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: <AppFuture at 0x152625ba7210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 333a8ae518d75b7527eaabcfad3b30e3
parsl.dataflow.memoization INFO: Task 108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e081f529-8b12-4a11-9766-a2d820793e68_isrForCrosstalkSources_946686_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e081f529-8b12-4a11-9766-a2d820793e68_isrForCrosstalkSources_946686_36.stderr
parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: <AppFuture at 0x1526271a91d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 92e24ccef8ddf1516c212bc0684a8fa9
parsl.dataflow.memoization INFO: Task 109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9ba33c47-d693-4e83-b886-ad4ba28edce9_isrForCrosstalkSources_946686_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9ba33c47-d693-4e83-b886-ad4ba28edce9_isrForCrosstalkSources_946686_16.stderr
parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: <AppFuture at 0x152625bb8090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 20315949ba58223a12ac843593c4c337
parsl.dataflow.memoization INFO: Task 110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cc6be176-2672-40fd-ad51-0403cff0a50f_isrForCrosstalkSources_946686_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cc6be176-2672-40fd-ad51-0403cff0a50f_isrForCrosstalkSources_946686_56.stderr
parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: <AppFuture at 0x152625bb8210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 081de032835e4b45500f895c94f92173
parsl.dataflow.memoization INFO: Task 111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/028e5956-2dc0-4097-bfb4-855424cf7273_isrForCrosstalkSources_946687_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/028e5956-2dc0-4097-bfb4-855424cf7273_isrForCrosstalkSources_946687_49.stderr
parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: <AppFuture at 0x152625bba250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 173c555091129452df7447edf44ddf2e
parsl.dataflow.memoization INFO: Task 112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5ba0e475-1de7-44a1-83aa-4d2b26ae6615_isrForCrosstalkSources_946688_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5ba0e475-1de7-44a1-83aa-4d2b26ae6615_isrForCrosstalkSources_946688_6.stderr
parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: <AppFuture at 0x152625bbaa10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 753088b19649c85e0c025ccb45df48ca
parsl.dataflow.memoization INFO: Task 113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9110f5df-ac58-4dea-a34a-235fc6cb95c3_isrForCrosstalkSources_946686_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9110f5df-ac58-4dea-a34a-235fc6cb95c3_isrForCrosstalkSources_946686_46.stderr
parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: <AppFuture at 0x152625bbb4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 99826b06e503fd9426944ce60150b53a
parsl.dataflow.memoization INFO: Task 114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/77e0dce5-ea5a-4333-828b-8ebe0c004584_isrForCrosstalkSources_946686_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/77e0dce5-ea5a-4333-828b-8ebe0c004584_isrForCrosstalkSources_946686_33.stderr
parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: <AppFuture at 0x152625bbb750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 89ad1dff1ee3e8aabd3818f5fbcfd752
parsl.dataflow.memoization INFO: Task 115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/684a7899-b33c-4786-a945-3d9390da36ef_isrForCrosstalkSources_946686_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/684a7899-b33c-4786-a945-3d9390da36ef_isrForCrosstalkSources_946686_29.stderr
parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: <AppFuture at 0x152625ba44d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 116 has memoization hash a9f1be4a65eacb0e16299b54bd661bd0
parsl.dataflow.memoization INFO: Task 116 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/261055ab-7e69-4b6c-b5f1-e4c284cb69d5_isrForCrosstalkSources_946686_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/261055ab-7e69-4b6c-b5f1-e4c284cb69d5_isrForCrosstalkSources_946686_13.stderr
parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: <AppFuture at 0x152625bc4cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 117 has memoization hash fe3a77eb08b2ce468dea248783951081
parsl.dataflow.memoization INFO: Task 117 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/76a31f60-ac0f-4c9d-8fb3-274c772c2c3b_isrForCrosstalkSources_946688_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/76a31f60-ac0f-4c9d-8fb3-274c772c2c3b_isrForCrosstalkSources_946688_34.stderr
parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: <AppFuture at 0x152625bc5890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 0ef41390d7e33bbbc367b5d117923cfe
parsl.dataflow.memoization INFO: Task 118 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/42e13784-64ae-488a-8361-a6104196ece2_isrForCrosstalkSources_946686_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/42e13784-64ae-488a-8361-a6104196ece2_isrForCrosstalkSources_946686_10.stderr
parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: <AppFuture at 0x152625bc6450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 24b2bd357e840f427648708a0a415594
parsl.dataflow.memoization INFO: Task 119 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ef122ed1-5e4b-4cbc-899d-0620a1a19727_isrForCrosstalkSources_946687_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ef122ed1-5e4b-4cbc-899d-0620a1a19727_isrForCrosstalkSources_946687_21.stderr
parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: <AppFuture at 0x152625bc6a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 2b8378a2b9597a45c8e8f64cb0d8ae9f
parsl.dataflow.memoization INFO: Task 120 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e6ad83f3-9dde-4327-913b-05c762193fd4_isrForCrosstalkSources_946686_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e6ad83f3-9dde-4327-913b-05c762193fd4_isrForCrosstalkSources_946686_2.stderr
parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: <AppFuture at 0x152625bc77d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 5b2798adbfeffbdfc333c0d904a7116a
parsl.dataflow.memoization INFO: Task 121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/04f54793-46e2-4e26-b0c5-80633bba5215_isrForCrosstalkSources_946687_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/04f54793-46e2-4e26-b0c5-80633bba5215_isrForCrosstalkSources_946687_20.stderr
parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: <AppFuture at 0x152625bc6b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 0f3474f75ef279134a7c571e5199f635
parsl.dataflow.memoization INFO: Task 122 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4a970540-4a53-450d-8bb2-d0391790b2ce_isrForCrosstalkSources_946686_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4a970540-4a53-450d-8bb2-d0391790b2ce_isrForCrosstalkSources_946686_25.stderr
parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: <AppFuture at 0x152625bd1010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 123 has memoization hash f5fbcadbdffec1e69c81cf65671e58a6
parsl.dataflow.memoization INFO: Task 123 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/b5d2e7ce-476b-442e-8772-42fa001fb3ad_isrForCrosstalkSources_946686_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/b5d2e7ce-476b-442e-8772-42fa001fb3ad_isrForCrosstalkSources_946686_32.stderr
parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: <AppFuture at 0x152625bd1710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 404dc4d12d0fb9ac34514e75dc5489ed
parsl.dataflow.memoization INFO: Task 124 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a508b17e-df09-44c1-862b-f9b7236cef1a_isrForCrosstalkSources_946688_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a508b17e-df09-44c1-862b-f9b7236cef1a_isrForCrosstalkSources_946688_45.stderr
parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: <AppFuture at 0x152625bd1f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 125 has memoization hash f51c50a25565dc807fdc70afd28e67c6
parsl.dataflow.memoization INFO: Task 125 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/68b6a7eb-f048-47bb-8c92-9fb5348ca55b_isrForCrosstalkSources_946686_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/68b6a7eb-f048-47bb-8c92-9fb5348ca55b_isrForCrosstalkSources_946686_50.stderr
parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: <AppFuture at 0x152625bd1450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 9fc01054420ea7e5797a8d64596c59d8
parsl.dataflow.memoization INFO: Task 126 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/fc12b118-5184-40d0-b7ee-44535794b891_isrForCrosstalkSources_946686_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/fc12b118-5184-40d0-b7ee-44535794b891_isrForCrosstalkSources_946686_41.stderr
parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: <AppFuture at 0x152625bd2450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 127 has memoization hash feb0aefb9de80a23ef6c9b9f649b804d
parsl.dataflow.memoization INFO: Task 127 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/f8a0d411-b765-49cd-8ed2-332136ab5aac_isrForCrosstalkSources_946688_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/f8a0d411-b765-49cd-8ed2-332136ab5aac_isrForCrosstalkSources_946688_49.stderr
parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: <AppFuture at 0x152625bd3010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 128 has memoization hash fe251734458ee8ba075368c962cf9503
parsl.dataflow.memoization INFO: Task 128 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c9aa980c-237b-49f9-9c88-47eacc2ea163_isrForCrosstalkSources_946688_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c9aa980c-237b-49f9-9c88-47eacc2ea163_isrForCrosstalkSources_946688_23.stderr
parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: <AppFuture at 0x152625bd35d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 129 has memoization hash f9068958fb5e802fcf37bf9cada3a401
parsl.dataflow.memoization INFO: Task 129 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a28e9b7f-7026-4224-bda4-5fdfd6ff1618_isrForCrosstalkSources_946688_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a28e9b7f-7026-4224-bda4-5fdfd6ff1618_isrForCrosstalkSources_946688_33.stderr
parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: <AppFuture at 0x152625bd3e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 686202d854aea0c79c4345c0b4abb20e
parsl.dataflow.memoization INFO: Task 130 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/a7f8f13e-bf1e-48fc-93da-38cc8040c161_isrForCrosstalkSources_946687_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/a7f8f13e-bf1e-48fc-93da-38cc8040c161_isrForCrosstalkSources_946687_14.stderr
parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: <AppFuture at 0x152625bd8ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 131 has memoization hash a756390e45168f686df402e5a4ce59d6
parsl.dataflow.memoization INFO: Task 131 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/76002a0b-03b2-4a44-bc98-804206e3a92c_isrForCrosstalkSources_946688_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/76002a0b-03b2-4a44-bc98-804206e3a92c_isrForCrosstalkSources_946688_29.stderr
parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: <AppFuture at 0x152625bda110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 132 has memoization hash b9de51b00b4e87aac683297b28eed697
parsl.dataflow.memoization INFO: Task 132 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ce3c2fef-3079-4119-bae3-e67f8b89126e_isrForCrosstalkSources_946687_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ce3c2fef-3079-4119-bae3-e67f8b89126e_isrForCrosstalkSources_946687_22.stderr
parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: <AppFuture at 0x152625bd9e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 1a24348acbf0640085102a15bd0cd0dd
parsl.dataflow.memoization INFO: Task 133 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d0aee24e-778b-4464-9f1b-dae92dd20016_isrForCrosstalkSources_946686_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d0aee24e-778b-4464-9f1b-dae92dd20016_isrForCrosstalkSources_946686_24.stderr
parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: <AppFuture at 0x152625bda510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 1116f3a459e5d15d4ca9d4b0d8bfd40d
parsl.dataflow.memoization INFO: Task 134 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/9ca6afb7-b109-4e9e-bc84-82bbecee5457_isrForCrosstalkSources_946687_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/9ca6afb7-b109-4e9e-bc84-82bbecee5457_isrForCrosstalkSources_946687_6.stderr
parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: <AppFuture at 0x152625bda890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 19f2c760286e31da20d6fb8aafd3f74c
parsl.dataflow.memoization INFO: Task 135 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/7caa6ccb-9495-4188-a21d-09c27a784711_isrForCrosstalkSources_946687_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/7caa6ccb-9495-4188-a21d-09c27a784711_isrForCrosstalkSources_946687_57.stderr
parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: <AppFuture at 0x152625bdb1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 34d656abd729ffd830ad41f29ce84356
parsl.dataflow.memoization INFO: Task 136 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/8dc284f1-d660-4099-8eea-9623102ed087_isrForCrosstalkSources_946686_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/8dc284f1-d660-4099-8eea-9623102ed087_isrForCrosstalkSources_946686_52.stderr
parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: <AppFuture at 0x152625bdb390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 137 has memoization hash fba66a8e4ea872f2df79b54f2540c177
parsl.dataflow.memoization INFO: Task 137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/6e308271-db08-4e4c-948b-6719447ed6f8_isrForCrosstalkSources_946687_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/6e308271-db08-4e4c-948b-6719447ed6f8_isrForCrosstalkSources_946687_46.stderr
parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: <AppFuture at 0x15262719d350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 05383d00f8a2df02d49660185ad1a293
parsl.dataflow.memoization INFO: Task 138 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e0cd7943-98ef-4f0f-b1bd-2be500dd6cc4_isrForCrosstalkSources_946688_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e0cd7943-98ef-4f0f-b1bd-2be500dd6cc4_isrForCrosstalkSources_946688_30.stderr
parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: <AppFuture at 0x152625be5bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 57a23823cc62562d830ad447f06d8ffc
parsl.dataflow.memoization INFO: Task 139 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/2347aa63-693f-4538-b98a-77fef22c75ac_isrForCrosstalkSources_946687_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/2347aa63-693f-4538-b98a-77fef22c75ac_isrForCrosstalkSources_946687_8.stderr
parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: <AppFuture at 0x152625be5850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 8b56a6abf4ea80345705a2d9fbce56c2
parsl.dataflow.memoization INFO: Task 140 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/fc674419-f3a7-4962-8187-6d495da7f427_isrForCrosstalkSources_946688_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/fc674419-f3a7-4962-8187-6d495da7f427_isrForCrosstalkSources_946688_9.stderr
parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: <AppFuture at 0x152625be5d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 141 has memoization hash bd84cacd31b840ac04539fcf3e05916f
parsl.dataflow.memoization INFO: Task 141 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/efb61d2f-14e1-4d7f-b11a-a1183da13343_isrForCrosstalkSources_946688_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/efb61d2f-14e1-4d7f-b11a-a1183da13343_isrForCrosstalkSources_946688_42.stderr
parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: <AppFuture at 0x152625be6c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 9c97354caa14178014e89e2853952725
parsl.dataflow.memoization INFO: Task 142 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4e7da617-51ac-4aa1-a991-52e987ab445c_isrForCrosstalkSources_946686_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4e7da617-51ac-4aa1-a991-52e987ab445c_isrForCrosstalkSources_946686_8.stderr
parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: <AppFuture at 0x152625be7650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 143 has memoization hash e867b2cf13082b5192f6384fa20490d6
parsl.dataflow.memoization INFO: Task 143 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/c13543fa-6197-4a7b-a01e-682a7b6b3374_isrForCrosstalkSources_946687_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/c13543fa-6197-4a7b-a01e-682a7b6b3374_isrForCrosstalkSources_946687_29.stderr
parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: <AppFuture at 0x152625be7a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 144 has memoization hash cf8cdb4d46c5d1170d4eb27f740c6edc
parsl.dataflow.memoization INFO: Task 144 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/8c629aae-66b6-4f30-8390-613c955e99dd_isrForCrosstalkSources_946687_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/8c629aae-66b6-4f30-8390-613c955e99dd_isrForCrosstalkSources_946687_16.stderr
parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: <AppFuture at 0x152625be7850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 145 has memoization hash b0c3f9665d9d09282e8f15ff7d327712
parsl.dataflow.memoization INFO: Task 145 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/03bfaf7c-eae5-4491-88ce-c0cfabab36fb_isrForCrosstalkSources_946687_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/03bfaf7c-eae5-4491-88ce-c0cfabab36fb_isrForCrosstalkSources_946687_48.stderr
parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: <AppFuture at 0x152625be9710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 9cf034bc90f5233315d8a4c44f0402d6
parsl.dataflow.memoization INFO: Task 146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c86bf880-bbe9-4309-beb4-37619754a50e_isrForCrosstalkSources_946688_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c86bf880-bbe9-4309-beb4-37619754a50e_isrForCrosstalkSources_946688_38.stderr
parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: <AppFuture at 0x152625be9590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 147 has memoization hash baf343639abe174033561ce325a71173
parsl.dataflow.memoization INFO: Task 147 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fc14254b-952d-4e54-b4c1-cdbcdcd55eac_isrForCrosstalkSources_946687_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fc14254b-952d-4e54-b4c1-cdbcdcd55eac_isrForCrosstalkSources_946687_33.stderr
parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: <AppFuture at 0x152625bea110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 1b2fc8ac86364b9050e682d2db2e2ec4
parsl.dataflow.memoization INFO: Task 148 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/624b7d24-759b-4604-a525-e4039a17d757_isrForCrosstalkSources_946688_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/624b7d24-759b-4604-a525-e4039a17d757_isrForCrosstalkSources_946688_40.stderr
parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: <AppFuture at 0x152625bead10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 1f096cba13f5eabb69000835e948812e
parsl.dataflow.memoization INFO: Task 149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/2a227480-3a94-48a8-9f4a-23b19b97944d_isrForCrosstalkSources_946686_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/2a227480-3a94-48a8-9f4a-23b19b97944d_isrForCrosstalkSources_946686_19.stderr
parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: <AppFuture at 0x152625bead50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 11e047215bad19db0ac571b075a21de5
parsl.dataflow.memoization INFO: Task 150 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ee017c49-940c-4023-8b86-4ce20eaba048_isrForCrosstalkSources_946687_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ee017c49-940c-4023-8b86-4ce20eaba048_isrForCrosstalkSources_946687_59.stderr
parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: <AppFuture at 0x152625be9a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 44e31a1cb007a6718a1f0e79a54e590f
parsl.dataflow.memoization INFO: Task 151 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5856952f-49b9-4161-be95-635cae78fc7b_isrForCrosstalkSources_946688_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5856952f-49b9-4161-be95-635cae78fc7b_isrForCrosstalkSources_946688_4.stderr
parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: <AppFuture at 0x152625bebb10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 152 has memoization hash c85dd5da6bc6f00ca41a56be8e9b50d4
parsl.dataflow.memoization INFO: Task 152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/07679853-515e-4df7-8deb-fa4f233889e8_isrForCrosstalkSources_946687_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/07679853-515e-4df7-8deb-fa4f233889e8_isrForCrosstalkSources_946687_43.stderr
parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: <AppFuture at 0x152625be4dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 7a3647143d3e21eb8f4a63c4fb5b0283
parsl.dataflow.memoization INFO: Task 153 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/76aef661-2b0b-4b97-93b8-c329408dc6d1_isrForCrosstalkSources_946687_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/76aef661-2b0b-4b97-93b8-c329408dc6d1_isrForCrosstalkSources_946687_13.stderr
parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: <AppFuture at 0x152625bf5ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 630992000806ef72be18ba686f4d3311
parsl.dataflow.memoization INFO: Task 154 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/6fd216ee-8e2d-4a57-8779-2ab64ee376d7_isrForCrosstalkSources_946686_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/6fd216ee-8e2d-4a57-8779-2ab64ee376d7_isrForCrosstalkSources_946686_61.stderr
parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: <AppFuture at 0x152625bf5750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 38682bbb9466280b8e3fd279a51453df
parsl.dataflow.memoization INFO: Task 155 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/48e595e0-3d82-4213-a216-824a0e43f103_isrForCrosstalkSources_946688_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/48e595e0-3d82-4213-a216-824a0e43f103_isrForCrosstalkSources_946688_43.stderr
parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: <AppFuture at 0x152625bf5b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 00c7afb7246fc99187fa3331990cb09c
parsl.dataflow.memoization INFO: Task 156 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4c1ca469-decb-4dec-906a-45ee8108ba3c_isrForCrosstalkSources_946687_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4c1ca469-decb-4dec-906a-45ee8108ba3c_isrForCrosstalkSources_946687_18.stderr
parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: <AppFuture at 0x152625be6fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 157 has memoization hash ebdf68307ec2be7913ffa2c65ef57734
parsl.dataflow.memoization INFO: Task 157 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/905511de-fb3a-41f6-baa8-10b6c29d6bd9_isrForCrosstalkSources_946686_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/905511de-fb3a-41f6-baa8-10b6c29d6bd9_isrForCrosstalkSources_946686_37.stderr
parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: <AppFuture at 0x152625bf7110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 1ee5da1dc446cd66316c0780cd603949
parsl.dataflow.memoization INFO: Task 158 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/9269d608-b29b-4e79-bf46-674008eb09c3_isrForCrosstalkSources_946688_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/9269d608-b29b-4e79-bf46-674008eb09c3_isrForCrosstalkSources_946688_50.stderr
parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: <AppFuture at 0x152625bf7910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 159 has memoization hash f517c9a7c68adc4687e3e3b9036e64a0
parsl.dataflow.memoization INFO: Task 159 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/feb544b1-b9fc-48d5-8bae-204432d19eca_isrForCrosstalkSources_946687_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/feb544b1-b9fc-48d5-8bae-204432d19eca_isrForCrosstalkSources_946687_32.stderr
parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: <AppFuture at 0x152625bf6dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 160 has memoization hash fc74ce6f4ace2297f76d70535250e252
parsl.dataflow.memoization INFO: Task 160 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/974722d3-caae-48d8-8d18-c010c322b8cc_isrForCrosstalkSources_946688_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/974722d3-caae-48d8-8d18-c010c322b8cc_isrForCrosstalkSources_946688_57.stderr
parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: <AppFuture at 0x152627100dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 161 has memoization hash dc80d39a258c3a38e4aefb93cd6d0ed4
parsl.dataflow.memoization INFO: Task 161 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/c3147434-7a6f-4c0e-a883-94c166bf5f12_isrForCrosstalkSources_946686_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/c3147434-7a6f-4c0e-a883-94c166bf5f12_isrForCrosstalkSources_946686_11.stderr
parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: <AppFuture at 0x152625c09410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 3de4dc9a68cb93979424672ca06c754d
parsl.dataflow.memoization INFO: Task 162 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/23e114d7-93bd-44f6-94e9-3d22d5d2fa00_isrForCrosstalkSources_946686_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/23e114d7-93bd-44f6-94e9-3d22d5d2fa00_isrForCrosstalkSources_946686_7.stderr
parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: <AppFuture at 0x152625c0a110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 4ffe71a2cfe520cf082daed1c6c69f4c
parsl.dataflow.memoization INFO: Task 163 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8639ed1d-5a4c-4408-9709-b1e6f22f3b5b_isrForCrosstalkSources_946688_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8639ed1d-5a4c-4408-9709-b1e6f22f3b5b_isrForCrosstalkSources_946688_28.stderr
parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: <AppFuture at 0x152625c09c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 19a361e4df5989f78f168d53152cbe8a
parsl.dataflow.memoization INFO: Task 164 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/fc03c504-c44c-4f92-a148-20cc496aaeef_isrForCrosstalkSources_946686_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/fc03c504-c44c-4f92-a148-20cc496aaeef_isrForCrosstalkSources_946686_47.stderr
parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: <AppFuture at 0x152625c0b150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 165 has memoization hash e20378d25cfe362cb1db4100b1172598
parsl.dataflow.memoization INFO: Task 165 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/27feb194-1822-4d31-9d11-867cfe20a62f_isrForCrosstalkSources_946687_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/27feb194-1822-4d31-9d11-867cfe20a62f_isrForCrosstalkSources_946687_44.stderr
parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: <AppFuture at 0x152625c0b7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 67f14207f12887210e9ee47da454bbd3
parsl.dataflow.memoization INFO: Task 166 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/168a4ecd-e58d-4f74-ab0a-4872cd76f81b_isrForCrosstalkSources_946688_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/168a4ecd-e58d-4f74-ab0a-4872cd76f81b_isrForCrosstalkSources_946688_52.stderr
parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: <AppFuture at 0x152625c0b110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 4782818124f9f7291d6e9c2fab8e084e
parsl.dataflow.memoization INFO: Task 167 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4fb831f6-ed95-4aad-bd49-fd2940b15e46_isrForCrosstalkSources_946687_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4fb831f6-ed95-4aad-bd49-fd2940b15e46_isrForCrosstalkSources_946687_24.stderr
parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: <AppFuture at 0x152625bc4e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 168 has memoization hash f2aa8f873681dc28364227f81d0c2258
parsl.dataflow.memoization INFO: Task 168 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/56d6ea46-1589-4853-9a03-78238503cd14_isrForCrosstalkSources_946687_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/56d6ea46-1589-4853-9a03-78238503cd14_isrForCrosstalkSources_946687_38.stderr
parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: <AppFuture at 0x152625c0d810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 26d6b29c0a1c5e1803ca25db34867149
parsl.dataflow.memoization INFO: Task 169 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a71c4856-64ea-47a4-8ba4-2f1d3f09627e_isrForCrosstalkSources_946688_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a71c4856-64ea-47a4-8ba4-2f1d3f09627e_isrForCrosstalkSources_946688_37.stderr
parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: <AppFuture at 0x152625c0d650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 3c0947eaf9206710a7290b3dd53634e6
parsl.dataflow.memoization INFO: Task 170 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fcc109fb-d4ab-4915-a7a9-7f65ab131aea_isrForCrosstalkSources_946687_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fcc109fb-d4ab-4915-a7a9-7f65ab131aea_isrForCrosstalkSources_946687_39.stderr
parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: <AppFuture at 0x152625c0da10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 9fcb9ba7f9b8692fe9553715f449023d
parsl.dataflow.memoization INFO: Task 171 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9e20474d-0e3c-416f-b881-c275652fac32_isrForCrosstalkSources_946686_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9e20474d-0e3c-416f-b881-c275652fac32_isrForCrosstalkSources_946686_43.stderr
parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: <AppFuture at 0x152625c0e9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 41c8a168f641767111ef08e1533e4858
parsl.dataflow.memoization INFO: Task 172 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/87e94383-1fd3-4a08-b337-b8699219a576_isrForCrosstalkSources_946686_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/87e94383-1fd3-4a08-b337-b8699219a576_isrForCrosstalkSources_946686_3.stderr
parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: <AppFuture at 0x152625c0f190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 173 has memoization hash f947c8ac3a34c3e577b8a9aae634d497
parsl.dataflow.memoization INFO: Task 173 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/d3d59d75-2113-4c05-8825-49dcef9e797e_isrForCrosstalkSources_946688_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/d3d59d75-2113-4c05-8825-49dcef9e797e_isrForCrosstalkSources_946688_56.stderr
parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: <AppFuture at 0x152625c0eed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 1644b896c8dbfe00e1a27b20bd88a994
parsl.dataflow.memoization INFO: Task 174 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cd68e496-0d53-46a1-a35b-3f3a00592ca1_isrForCrosstalkSources_946686_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cd68e496-0d53-46a1-a35b-3f3a00592ca1_isrForCrosstalkSources_946686_55.stderr
parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: <AppFuture at 0x152625c0fcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 175 has memoization hash f29d614fc518c68b9f00f7fcce1cac4b
parsl.dataflow.memoization INFO: Task 175 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8a50bb83-c024-4afd-b704-2a1554f721d4_isrForCrosstalkSources_946688_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8a50bb83-c024-4afd-b704-2a1554f721d4_isrForCrosstalkSources_946688_35.stderr
parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: <AppFuture at 0x152625c145d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 0a6436f201ff75f2284a99658e0303bd
parsl.dataflow.memoization INFO: Task 176 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ddd3844e-a96e-4092-aed5-44a699f3cf19_isrForCrosstalkSources_946686_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ddd3844e-a96e-4092-aed5-44a699f3cf19_isrForCrosstalkSources_946686_15.stderr
parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: <AppFuture at 0x152625c15290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 9d51086c96dbe6d25743823a69dd1a1e
parsl.dataflow.memoization INFO: Task 177 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/03149966-e2a6-4c9f-86c8-ef2562be6cea_isrForCrosstalkSources_946686_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/03149966-e2a6-4c9f-86c8-ef2562be6cea_isrForCrosstalkSources_946686_35.stderr
parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: <AppFuture at 0x152625c15b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 32d6c3a5d8e6a4389b90ec98c421e026
parsl.dataflow.memoization INFO: Task 178 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/132ba0b4-6981-44ef-926e-17789d95e467_isrForCrosstalkSources_946688_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/132ba0b4-6981-44ef-926e-17789d95e467_isrForCrosstalkSources_946688_21.stderr
parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: <AppFuture at 0x152625c16610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 055207fb568aba35b008e6d51b6a2722
parsl.dataflow.memoization INFO: Task 179 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/05ab0909-541f-4b18-99e4-da5f1bd47723_isrForCrosstalkSources_946688_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/05ab0909-541f-4b18-99e4-da5f1bd47723_isrForCrosstalkSources_946688_53.stderr
parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: <AppFuture at 0x152625c16c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 180 has memoization hash ab3aff1758655d31da9f958025035af9
parsl.dataflow.memoization INFO: Task 180 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9a8a2113-5b7c-4070-85d1-921543b44e41_isrForCrosstalkSources_946686_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9a8a2113-5b7c-4070-85d1-921543b44e41_isrForCrosstalkSources_946686_49.stderr
parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: <AppFuture at 0x152625c17510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 181 has memoization hash b7e53177bcc4adb8575613c9bfa71ba0
parsl.dataflow.memoization INFO: Task 181 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fcf2f95e-37f9-4364-b910-67335135bc87_isrForCrosstalkSources_946687_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fcf2f95e-37f9-4364-b910-67335135bc87_isrForCrosstalkSources_946687_10.stderr
parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: <AppFuture at 0x152625c17c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 52e0e29e7d76106a39d93449e32fcd3c
parsl.dataflow.memoization INFO: Task 182 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/b119e2ed-110d-4301-8d42-1454bd5fc7be_isrForCrosstalkSources_946687_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/b119e2ed-110d-4301-8d42-1454bd5fc7be_isrForCrosstalkSources_946687_58.stderr
parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: <AppFuture at 0x152625c14fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 168375e124aa1a36654af49dbbca64bc
parsl.dataflow.memoization INFO: Task 183 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b1510b04-9cb1-49b3-a80f-2054163a6c2f_isrForCrosstalkSources_946688_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b1510b04-9cb1-49b3-a80f-2054163a6c2f_isrForCrosstalkSources_946688_51.stderr
parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: <AppFuture at 0x152625c21710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 206dd764be13139002b2dc9c9cd02f47
parsl.dataflow.memoization INFO: Task 184 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/c4265395-b27a-458b-b2e9-7e9d3688f16d_isrForCrosstalkSources_946686_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/c4265395-b27a-458b-b2e9-7e9d3688f16d_isrForCrosstalkSources_946686_44.stderr
parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: <AppFuture at 0x152625c21610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 9015e25b80b7865095ec047b61375feb
parsl.dataflow.memoization INFO: Task 185 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x152625bb31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/17ce4d14-a907-4197-8fdc-ca271b45ff3b_isrForCrosstalkSources_946688_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/17ce4d14-a907-4197-8fdc-ca271b45ff3b_isrForCrosstalkSources_946688_17.stderr
2024-03-14 02:51:18 proc_lsst.multi:146 [INFO]  found job 13859 in provider local
proc_lsst.multi INFO: found job 13859 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 186
parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism
parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out
parsl.dataflow.strategy DEBUG: Requesting 46 more blocks
parsl.executors.status_handling INFO: Scaling out by 46 blocks
parsl.executors.status_handling INFO: Allocated block ID 1
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1
2024-03-14 02:51:18 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
2024-03-14 02:51:18 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:18 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-14 02:51:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:18 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:19 proc_lsst.multi:170 [INFO]  [multi] job_id 17028877
proc_lsst.multi INFO: [multi] job_id 17028877
2024-03-14 02:51:19 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:19 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:19 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028877
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028877
parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17028877
parsl.executors.status_handling INFO: Allocated block ID 2
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2
2024-03-14 02:51:19 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
2024-03-14 02:51:19 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:19 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:19 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:19 proc_lsst.multi:170 [INFO]  [multi] job_id 17028878
proc_lsst.multi INFO: [multi] job_id 17028878
2024-03-14 02:51:19 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 02:51:19 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:19 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028878
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028878
parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17028878
parsl.executors.status_handling INFO: Allocated block ID 3
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3
2024-03-14 02:51:19 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
2024-03-14 02:51:19 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:19 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:19 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 02:51:19 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:19 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:20 proc_lsst.multi:170 [INFO]  [multi] job_id 17028879
proc_lsst.multi INFO: [multi] job_id 17028879
2024-03-14 02:51:20 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 02:51:20 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:20 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028879
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028879
parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17028879
parsl.executors.status_handling INFO: Allocated block ID 4
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4
2024-03-14 02:51:20 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
2024-03-14 02:51:20 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:20 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 02:51:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:20 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:20 proc_lsst.multi:170 [INFO]  [multi] job_id 17028880
proc_lsst.multi INFO: [multi] job_id 17028880
2024-03-14 02:51:20 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 02:51:20 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:20 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028880
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028880
parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17028880
parsl.executors.status_handling INFO: Allocated block ID 5
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5
2024-03-14 02:51:20 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
2024-03-14 02:51:20 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:20 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 02:51:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:20 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:20 proc_lsst.multi:170 [INFO]  [multi] job_id 17028881
proc_lsst.multi INFO: [multi] job_id 17028881
2024-03-14 02:51:20 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:20 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:20 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028881
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028881
parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17028881
parsl.executors.status_handling INFO: Allocated block ID 6
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6
2024-03-14 02:51:20 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
2024-03-14 02:51:20 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:20 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:20 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:21 proc_lsst.multi:170 [INFO]  [multi] job_id 17028882
proc_lsst.multi INFO: [multi] job_id 17028882
2024-03-14 02:51:21 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 02:51:21 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:21 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028882
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028882
parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17028882
parsl.executors.status_handling INFO: Allocated block ID 7
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7
2024-03-14 02:51:21 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
2024-03-14 02:51:21 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:21 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 02:51:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:21 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:21 proc_lsst.multi:170 [INFO]  [multi] job_id 17028883
proc_lsst.multi INFO: [multi] job_id 17028883
2024-03-14 02:51:21 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 02:51:21 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:21 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028883
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028883
parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17028883
parsl.executors.status_handling INFO: Allocated block ID 8
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8
2024-03-14 02:51:21 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
2024-03-14 02:51:21 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:21 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 02:51:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:21 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:21 proc_lsst.multi:170 [INFO]  [multi] job_id 17028884
proc_lsst.multi INFO: [multi] job_id 17028884
2024-03-14 02:51:21 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 02:51:21 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:21 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028884
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028884
parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17028884
parsl.executors.status_handling INFO: Allocated block ID 9
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9
2024-03-14 02:51:21 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
2024-03-14 02:51:21 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:21 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:21 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 02:51:21 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:21 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:22 proc_lsst.multi:170 [INFO]  [multi] job_id 17028885
proc_lsst.multi INFO: [multi] job_id 17028885
2024-03-14 02:51:22 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 02:51:22 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:22 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028885
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028885
parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17028885
parsl.executors.status_handling INFO: Allocated block ID 10
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10
2024-03-14 02:51:22 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
2024-03-14 02:51:22 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:22 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 02:51:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:22 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:23 proc_lsst.multi:170 [INFO]  [multi] job_id 17028886
proc_lsst.multi INFO: [multi] job_id 17028886
2024-03-14 02:51:23 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 02:51:23 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:23 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028886
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028886
parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17028886
parsl.executors.status_handling INFO: Allocated block ID 11
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11
2024-03-14 02:51:23 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
2024-03-14 02:51:23 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:23 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 02:51:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:23 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:24 proc_lsst.multi:170 [INFO]  [multi] job_id 17028887
proc_lsst.multi INFO: [multi] job_id 17028887
2024-03-14 02:51:24 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-14 02:51:24 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:24 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028887
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028887
parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17028887
parsl.executors.status_handling INFO: Allocated block ID 12
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12
2024-03-14 02:51:24 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
2024-03-14 02:51:24 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:24 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-14 02:51:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:24 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:25 proc_lsst.multi:170 [INFO]  [multi] job_id 17028888
proc_lsst.multi INFO: [multi] job_id 17028888
2024-03-14 02:51:25 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-14 02:51:25 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:25 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028888
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028888
parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17028888
parsl.executors.status_handling INFO: Allocated block ID 13
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13
2024-03-14 02:51:25 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
2024-03-14 02:51:25 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:25 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-14 02:51:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:25 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:25 proc_lsst.multi:170 [INFO]  [multi] job_id 17028889
proc_lsst.multi INFO: [multi] job_id 17028889
2024-03-14 02:51:25 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-14 02:51:25 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:25 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028889
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028889
parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17028889
parsl.executors.status_handling INFO: Allocated block ID 14
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14
2024-03-14 02:51:25 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
2024-03-14 02:51:25 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:25 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-14 02:51:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:25 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:26 proc_lsst.multi:170 [INFO]  [multi] job_id 17028890
proc_lsst.multi INFO: [multi] job_id 17028890
2024-03-14 02:51:26 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-14 02:51:26 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:26 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028890
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028890
parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17028890
parsl.executors.status_handling INFO: Allocated block ID 15
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15
2024-03-14 02:51:26 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
2024-03-14 02:51:26 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:26 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-14 02:51:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:26 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:27 proc_lsst.multi:170 [INFO]  [multi] job_id 17028891
proc_lsst.multi INFO: [multi] job_id 17028891
2024-03-14 02:51:27 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-14 02:51:27 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:27 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028891
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028891
parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17028891
parsl.executors.status_handling INFO: Allocated block ID 16
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16
2024-03-14 02:51:27 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
2024-03-14 02:51:27 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:27 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-14 02:51:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:27 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:28 proc_lsst.multi:170 [INFO]  [multi] job_id 17028892
proc_lsst.multi INFO: [multi] job_id 17028892
2024-03-14 02:51:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-14 02:51:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:28 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028892
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028892
parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17028892
parsl.executors.status_handling INFO: Allocated block ID 17
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17
2024-03-14 02:51:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
2024-03-14 02:51:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-14 02:51:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:28 proc_lsst.multi:170 [INFO]  [multi] job_id 17028893
proc_lsst.multi INFO: [multi] job_id 17028893
2024-03-14 02:51:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-14 02:51:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:28 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028893
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028893
parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17028893
parsl.executors.status_handling INFO: Allocated block ID 18
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18
2024-03-14 02:51:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
2024-03-14 02:51:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-14 02:51:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:28 proc_lsst.multi:170 [INFO]  [multi] job_id 17028894
proc_lsst.multi INFO: [multi] job_id 17028894
2024-03-14 02:51:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-14 02:51:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:28 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028894
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028894
parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17028894
parsl.executors.status_handling INFO: Allocated block ID 19
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19
2024-03-14 02:51:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
2024-03-14 02:51:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-14 02:51:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17028895
proc_lsst.multi INFO: [multi] job_id 17028895
2024-03-14 02:51:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-14 02:51:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:29 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028895
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028895
parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17028895
parsl.executors.status_handling INFO: Allocated block ID 20
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20
2024-03-14 02:51:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
2024-03-14 02:51:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-14 02:51:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17028896
proc_lsst.multi INFO: [multi] job_id 17028896
2024-03-14 02:51:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 02:51:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:29 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028896
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028896
parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17028896
parsl.executors.status_handling INFO: Allocated block ID 21
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21
2024-03-14 02:51:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
2024-03-14 02:51:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-14 02:51:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17028897
proc_lsst.multi INFO: [multi] job_id 17028897
2024-03-14 02:51:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-14 02:51:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:30 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028897
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028897
parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17028897
parsl.executors.status_handling INFO: Allocated block ID 22
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22
2024-03-14 02:51:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
2024-03-14 02:51:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-14 02:51:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17028898
proc_lsst.multi INFO: [multi] job_id 17028898
2024-03-14 02:51:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-14 02:51:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:30 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028898
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028898
parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17028898
parsl.executors.status_handling INFO: Allocated block ID 23
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23
2024-03-14 02:51:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
2024-03-14 02:51:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-14 02:51:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17028899
proc_lsst.multi INFO: [multi] job_id 17028899
2024-03-14 02:51:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-14 02:51:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:30 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028899
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028899
parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17028899
parsl.executors.status_handling INFO: Allocated block ID 24
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24
2024-03-14 02:51:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
2024-03-14 02:51:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-14 02:51:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17028900
proc_lsst.multi INFO: [multi] job_id 17028900
2024-03-14 02:51:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-14 02:51:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:30 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028900
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028900
parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17028900
parsl.executors.status_handling INFO: Allocated block ID 25
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25
2024-03-14 02:51:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
2024-03-14 02:51:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-14 02:51:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:31 proc_lsst.multi:170 [INFO]  [multi] job_id 17028901
proc_lsst.multi INFO: [multi] job_id 17028901
2024-03-14 02:51:31 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-14 02:51:31 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:31 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028901
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028901
parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17028901
parsl.executors.status_handling INFO: Allocated block ID 26
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26
2024-03-14 02:51:31 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
2024-03-14 02:51:31 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:31 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-14 02:51:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:31 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 86fae69a7ffe77c8150cfd565c07c623 with result from task 0
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:31 proc_lsst.multi:170 [INFO]  [multi] job_id 17028902
proc_lsst.multi INFO: [multi] job_id 17028902
2024-03-14 02:51:31 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-14 02:51:31 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:31 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028902
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028902
parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17028902
parsl.executors.status_handling INFO: Allocated block ID 27
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27
2024-03-14 02:51:31 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
2024-03-14 02:51:31 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:31 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-14 02:51:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:31 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow DEBUG: Task 0 checkpointed
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/178acbf3-feef-467d-a18f-55a4173b7e0f_isrForCrosstalkSources_946686_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/178acbf3-feef-467d-a18f-55a4173b7e0f_isrForCrosstalkSources_946686_30.stderr
parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0270d8e774767babec74314ae0ddb81e with result from task 1
parsl.dataflow.dflow DEBUG: Task 1 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/a224889e-50f9-461b-9ac6-201084ef6afe_isrForCrosstalkSources_946687_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/a224889e-50f9-461b-9ac6-201084ef6afe_isrForCrosstalkSources_946687_45.stderr
2024-03-14 02:51:31 proc_lsst.multi:170 [INFO]  [multi] job_id 17028903
proc_lsst.multi INFO: [multi] job_id 17028903
2024-03-14 02:51:31 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-14 02:51:31 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:31 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028903
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028903
parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17028903
parsl.executors.status_handling INFO: Allocated block ID 28
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28
2024-03-14 02:51:31 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
2024-03-14 02:51:31 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:31 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-14 02:51:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:31 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:32 proc_lsst.multi:170 [INFO]  [multi] job_id 17028904
proc_lsst.multi INFO: [multi] job_id 17028904
2024-03-14 02:51:32 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-14 02:51:32 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:32 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028904
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028904
parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17028904
parsl.executors.status_handling INFO: Allocated block ID 29
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29
2024-03-14 02:51:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
2024-03-14 02:51:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:32 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-14 02:51:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:33 proc_lsst.multi:170 [INFO]  [multi] job_id 17028905
proc_lsst.multi INFO: [multi] job_id 17028905
2024-03-14 02:51:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-14 02:51:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:33 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028905
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028905
parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17028905
parsl.executors.status_handling INFO: Allocated block ID 30
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30
2024-03-14 02:51:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
2024-03-14 02:51:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-14 02:51:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:34 proc_lsst.multi:170 [INFO]  [multi] job_id 17028906
proc_lsst.multi INFO: [multi] job_id 17028906
2024-03-14 02:51:34 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:34 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:34 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17028906
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17028906
parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17028906
parsl.executors.status_handling INFO: Allocated block ID 31
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31
2024-03-14 02:51:34 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
2024-03-14 02:51:34 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:34 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:34 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:34 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-14 02:51:34 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:34 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:35 proc_lsst.multi:170 [INFO]  [multi] job_id 17028908
proc_lsst.multi INFO: [multi] job_id 17028908
2024-03-14 02:51:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:35 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17028908
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17028908
parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17028908
parsl.executors.status_handling INFO: Allocated block ID 32
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32
2024-03-14 02:51:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
2024-03-14 02:51:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:35 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:35 proc_lsst.multi:170 [INFO]  [multi] job_id 17028909
proc_lsst.multi INFO: [multi] job_id 17028909
2024-03-14 02:51:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 02:51:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:35 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17028909
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17028909
parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17028909
parsl.executors.status_handling INFO: Allocated block ID 33
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33
2024-03-14 02:51:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
2024-03-14 02:51:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:35 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 02:51:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17028910
proc_lsst.multi INFO: [multi] job_id 17028910
2024-03-14 02:51:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 02:51:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17028910
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17028910
parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17028910
parsl.executors.status_handling INFO: Allocated block ID 34
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34
2024-03-14 02:51:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
2024-03-14 02:51:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 02:51:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:38 proc_lsst.multi:170 [INFO]  [multi] job_id 17028911
proc_lsst.multi INFO: [multi] job_id 17028911
2024-03-14 02:51:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 02:51:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:38 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17028911
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17028911
parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17028911
parsl.executors.status_handling INFO: Allocated block ID 35
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35
2024-03-14 02:51:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
2024-03-14 02:51:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:38 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 02:51:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17028912
proc_lsst.multi INFO: [multi] job_id 17028912
2024-03-14 02:51:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:39 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17028912
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17028912
parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17028912
parsl.executors.status_handling INFO: Allocated block ID 36
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36
2024-03-14 02:51:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17028913
proc_lsst.multi INFO: [multi] job_id 17028913
2024-03-14 02:51:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:39 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028913
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028913
parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17028913
parsl.executors.status_handling INFO: Allocated block ID 37
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37
2024-03-14 02:51:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17028914
proc_lsst.multi INFO: [multi] job_id 17028914
2024-03-14 02:51:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 02:51:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:39 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028914
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028914
parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17028914
parsl.executors.status_handling INFO: Allocated block ID 38
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38
2024-03-14 02:51:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17028915
proc_lsst.multi INFO: [multi] job_id 17028915
2024-03-14 02:51:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 02:51:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:39 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028915
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028915
parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17028915
parsl.executors.status_handling INFO: Allocated block ID 39
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39
2024-03-14 02:51:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-14 02:51:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17028916
proc_lsst.multi INFO: [multi] job_id 17028916
2024-03-14 02:51:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 02:51:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028916
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028916
parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17028916
parsl.executors.status_handling INFO: Allocated block ID 40
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40
2024-03-14 02:51:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b46af84bfa8ab435cfca231e676bef4 with result from task 3
parsl.dataflow.dflow DEBUG: Task 3 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b03e98d8-bd45-4ebd-8328-57c283312bf2_isrForCrosstalkSources_946688_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b03e98d8-bd45-4ebd-8328-57c283312bf2_isrForCrosstalkSources_946688_47.stderr
2024-03-14 02:51:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17028917
proc_lsst.multi INFO: [multi] job_id 17028917
2024-03-14 02:51:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028917
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028917
parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17028917
parsl.executors.status_handling INFO: Allocated block ID 41
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41
2024-03-14 02:51:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17028918
proc_lsst.multi INFO: [multi] job_id 17028918
2024-03-14 02:51:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 02:51:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028918
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028918
parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17028918
parsl.executors.status_handling INFO: Allocated block ID 42
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42
2024-03-14 02:51:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d1c9300894989449c1cc0ee318c32c11 with result from task 2
parsl.dataflow.dflow DEBUG: Task 2 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ddd6f3f8-8249-4ca2-aca0-ad090a0a1cbb_isrForCrosstalkSources_946687_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ddd6f3f8-8249-4ca2-aca0-ad090a0a1cbb_isrForCrosstalkSources_946687_34.stderr
2024-03-14 02:51:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17028919
proc_lsst.multi INFO: [multi] job_id 17028919
2024-03-14 02:51:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 02:51:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028919
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028919
parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17028919
parsl.executors.status_handling INFO: Allocated block ID 43
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43
2024-03-14 02:51:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17028920
proc_lsst.multi INFO: [multi] job_id 17028920
2024-03-14 02:51:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 02:51:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028920
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028920
parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17028920
parsl.executors.status_handling INFO: Allocated block ID 44
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44
2024-03-14 02:51:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:40 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-14 02:51:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17028921
proc_lsst.multi INFO: [multi] job_id 17028921
2024-03-14 02:51:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 02:51:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:41 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028921
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028921
parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17028921
parsl.executors.status_handling INFO: Allocated block ID 45
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45
2024-03-14 02:51:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
2024-03-14 02:51:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-14 02:51:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17028922
proc_lsst.multi INFO: [multi] job_id 17028922
2024-03-14 02:51:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 02:51:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:41 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028922
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028922
parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17028922
parsl.executors.status_handling INFO: Allocated block ID 46
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46
2024-03-14 02:51:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
2024-03-14 02:51:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-14 02:51:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-14 02:51:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-14 02:51:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-14 02:51:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-14 02:51:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-14 02:51:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-14 02:51:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-14 02:51:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-14 02:51:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 10.64.65.9,n3009,169.254.95.120,10.64.129.9,127.0.0.1,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54572 --result_port=54031 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-14 02:51:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17028923
proc_lsst.multi INFO: [multi] job_id 17028923
2024-03-14 02:51:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-14 02:51:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-14 02:51:41 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17028923
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17028923
parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17028923
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 182
parsl.dataflow.strategy DEBUG: Executor multi has 182 active tasks, 1/46 running/pending blocks, and 14 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 182
parsl.dataflow.strategy DEBUG: Executor multi has 182 active tasks, 1/46 running/pending blocks, and 21 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ef30cb1a1286ab70643b256b6abc745 with result from task 8
parsl.dataflow.dflow DEBUG: Task 8 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cf5bf996-cf93-4a21-bcaf-7b7e8f202cdf_isrForCrosstalkSources_946686_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cf5bf996-cf93-4a21-bcaf-7b7e8f202cdf_isrForCrosstalkSources_946686_20.stderr
parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19072fe979f6d1aead4196cb6402d699 with result from task 11
parsl.dataflow.dflow DEBUG: Task 11 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d71f7cbe-7f4a-4419-a93e-78b197d7d723_isrForCrosstalkSources_946686_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d71f7cbe-7f4a-4419-a93e-78b197d7d723_isrForCrosstalkSources_946686_48.stderr
2024-03-14 02:51:51 proc_lsst.multi:146 [INFO]  found job 13859 in provider local
proc_lsst.multi INFO: found job 13859 in provider local
2024-03-14 02:51:51 proc_lsst.multi:146 [INFO]  found job 17028877 in provider astro
proc_lsst.multi INFO: found job 17028877 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:51 proc_lsst.multi:146 [INFO]  found job 17028878 in provider astro
proc_lsst.multi INFO: found job 17028878 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:51 proc_lsst.multi:146 [INFO]  found job 17028879 in provider astro
proc_lsst.multi INFO: found job 17028879 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:52 proc_lsst.multi:146 [INFO]  found job 17028880 in provider astro
proc_lsst.multi INFO: found job 17028880 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:52 proc_lsst.multi:146 [INFO]  found job 17028881 in provider astro
proc_lsst.multi INFO: found job 17028881 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:52 proc_lsst.multi:146 [INFO]  found job 17028882 in provider astro
proc_lsst.multi INFO: found job 17028882 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:53 proc_lsst.multi:146 [INFO]  found job 17028883 in provider astro
proc_lsst.multi INFO: found job 17028883 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:53 proc_lsst.multi:146 [INFO]  found job 17028884 in provider astro
proc_lsst.multi INFO: found job 17028884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:53 proc_lsst.multi:146 [INFO]  found job 17028885 in provider astro
proc_lsst.multi INFO: found job 17028885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:53 proc_lsst.multi:146 [INFO]  found job 17028886 in provider astro
proc_lsst.multi INFO: found job 17028886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:54 proc_lsst.multi:146 [INFO]  found job 17028887 in provider astro
proc_lsst.multi INFO: found job 17028887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:54 proc_lsst.multi:146 [INFO]  found job 17028888 in provider astro
proc_lsst.multi INFO: found job 17028888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:54 proc_lsst.multi:146 [INFO]  found job 17028889 in provider astro
proc_lsst.multi INFO: found job 17028889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:55 proc_lsst.multi:146 [INFO]  found job 17028890 in provider astro
proc_lsst.multi INFO: found job 17028890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:55 proc_lsst.multi:146 [INFO]  found job 17028891 in provider astro
proc_lsst.multi INFO: found job 17028891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:55 proc_lsst.multi:146 [INFO]  found job 17028892 in provider astro
proc_lsst.multi INFO: found job 17028892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 01e23e84ec20b8900d3ec3427d2677d7 with result from task 26
parsl.dataflow.dflow DEBUG: Task 26 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/19adc1b4-8230-46b3-bf95-adf328c99aa8_isrForCrosstalkSources_946687_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/19adc1b4-8230-46b3-bf95-adf328c99aa8_isrForCrosstalkSources_946687_40.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:55 proc_lsst.multi:146 [INFO]  found job 17028893 in provider astro
proc_lsst.multi INFO: found job 17028893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:56 proc_lsst.multi:146 [INFO]  found job 17028894 in provider astro
proc_lsst.multi INFO: found job 17028894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:56 proc_lsst.multi:146 [INFO]  found job 17028895 in provider astro
proc_lsst.multi INFO: found job 17028895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e83371ed0884d5e602226c3cbea9cf24 with result from task 25
parsl.dataflow.dflow DEBUG: Task 25 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/47ce4e8a-ed92-45bb-a03c-9ed357fa345a_isrForCrosstalkSources_946687_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/47ce4e8a-ed92-45bb-a03c-9ed357fa345a_isrForCrosstalkSources_946687_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:56 proc_lsst.multi:146 [INFO]  found job 17028896 in provider astro
proc_lsst.multi INFO: found job 17028896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:57 proc_lsst.multi:146 [INFO]  found job 17028897 in provider astro
proc_lsst.multi INFO: found job 17028897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:57 proc_lsst.multi:146 [INFO]  found job 17028898 in provider astro
proc_lsst.multi INFO: found job 17028898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:57 proc_lsst.multi:146 [INFO]  found job 17028899 in provider astro
proc_lsst.multi INFO: found job 17028899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:58 proc_lsst.multi:146 [INFO]  found job 17028900 in provider astro
proc_lsst.multi INFO: found job 17028900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:58 proc_lsst.multi:146 [INFO]  found job 17028901 in provider astro
proc_lsst.multi INFO: found job 17028901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:58 proc_lsst.multi:146 [INFO]  found job 17028902 in provider astro
proc_lsst.multi INFO: found job 17028902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:59 proc_lsst.multi:146 [INFO]  found job 17028903 in provider astro
proc_lsst.multi INFO: found job 17028903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:59 proc_lsst.multi:146 [INFO]  found job 17028904 in provider astro
proc_lsst.multi INFO: found job 17028904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:51:59 proc_lsst.multi:146 [INFO]  found job 17028905 in provider astro
proc_lsst.multi INFO: found job 17028905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:00 proc_lsst.multi:146 [INFO]  found job 17028906 in provider astro
proc_lsst.multi INFO: found job 17028906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:00 proc_lsst.multi:146 [INFO]  found job 17028908 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028908 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:00 proc_lsst.multi:146 [INFO]  found job 17028909 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028909 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:00 proc_lsst.multi:146 [INFO]  found job 17028910 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028910 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:00 proc_lsst.multi:146 [INFO]  found job 17028911 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028911 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:00 proc_lsst.multi:146 [INFO]  found job 17028912 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028912 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:00 proc_lsst.multi:146 [INFO]  found job 17028913 in provider escience_a40
proc_lsst.multi INFO: found job 17028913 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:01 proc_lsst.multi:146 [INFO]  found job 17028914 in provider escience_a40
proc_lsst.multi INFO: found job 17028914 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:01 proc_lsst.multi:146 [INFO]  found job 17028915 in provider escience_a40
proc_lsst.multi INFO: found job 17028915 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:01 proc_lsst.multi:146 [INFO]  found job 17028916 in provider escience_a40
proc_lsst.multi INFO: found job 17028916 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:02 proc_lsst.multi:146 [INFO]  found job 17028917 in provider escience_a40
proc_lsst.multi INFO: found job 17028917 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:02 proc_lsst.multi:146 [INFO]  found job 17028918 in provider escience_a40
proc_lsst.multi INFO: found job 17028918 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:02 proc_lsst.multi:146 [INFO]  found job 17028919 in provider escience_a40
proc_lsst.multi INFO: found job 17028919 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:03 proc_lsst.multi:146 [INFO]  found job 17028920 in provider escience_a40
proc_lsst.multi INFO: found job 17028920 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:03 proc_lsst.multi:146 [INFO]  found job 17028921 in provider escience_a40
proc_lsst.multi INFO: found job 17028921 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:03 proc_lsst.multi:146 [INFO]  found job 17028922 in provider escience_a40
proc_lsst.multi INFO: found job 17028922 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:04 proc_lsst.multi:146 [INFO]  found job 17028923 in provider escience_a40
proc_lsst.multi INFO: found job 17028923 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 178
parsl.dataflow.strategy DEBUG: Executor multi has 178 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 178
parsl.dataflow.strategy DEBUG: Executor multi has 178 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f66680b9d8425f47e9e6f34f9fc5ef93 with result from task 54
parsl.dataflow.dflow DEBUG: Task 54 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/14b273ed-8049-4d84-aaf4-d839ab050467_isrForCrosstalkSources_946686_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/14b273ed-8049-4d84-aaf4-d839ab050467_isrForCrosstalkSources_946686_23.stderr
parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a22024297b38593ced66f71b92407e8 with result from task 55
parsl.dataflow.dflow DEBUG: Task 55 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/02d9ac39-19b7-44f7-982e-f167a5dfc4de_isrForCrosstalkSources_946687_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/02d9ac39-19b7-44f7-982e-f167a5dfc4de_isrForCrosstalkSources_946687_55.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 176
parsl.dataflow.strategy DEBUG: Executor multi has 176 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02603638b1d3ddf38cab5c4a9bb49f0a with result from task 56
parsl.dataflow.dflow DEBUG: Task 56 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f39e930a-a837-45df-b8c8-a6a419c58962_isrForCrosstalkSources_946687_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f39e930a-a837-45df-b8c8-a6a419c58962_isrForCrosstalkSources_946687_35.stderr
parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b4126b6ae7efbf0a56254fc858838a1 with result from task 57
parsl.dataflow.dflow DEBUG: Task 57 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/8e88f2b6-07ae-4bc5-89e4-098d89ea164e_isrForCrosstalkSources_946687_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/8e88f2b6-07ae-4bc5-89e4-098d89ea164e_isrForCrosstalkSources_946687_25.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 174
parsl.dataflow.strategy DEBUG: Executor multi has 174 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f7396e907299f27d4425b46ab1c3f8d with result from task 12
parsl.dataflow.dflow DEBUG: Task 12 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e7b1e40b-4fc3-4f38-9630-e673fef8d429_isrForCrosstalkSources_946688_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e7b1e40b-4fc3-4f38-9630-e673fef8d429_isrForCrosstalkSources_946688_27.stderr
parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62b57d39b7d04cbc291a8465196997d6 with result from task 14
parsl.dataflow.dflow DEBUG: Task 14 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/7046e8a0-04e9-4e56-a7b3-9a7f2f7d3b5c_isrForCrosstalkSources_946687_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/7046e8a0-04e9-4e56-a7b3-9a7f2f7d3b5c_isrForCrosstalkSources_946687_27.stderr
parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b81ff6b798f72eba4828b2e4b8ca1298 with result from task 4
parsl.dataflow.dflow DEBUG: Task 4 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/3c30339b-870a-40c7-bf30-df3275bcbbbf_isrForCrosstalkSources_946688_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/3c30339b-870a-40c7-bf30-df3275bcbbbf_isrForCrosstalkSources_946688_22.stderr
parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc540036bd9df469063d82989d75d309 with result from task 7
parsl.dataflow.dflow DEBUG: Task 7 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/9f5dbf9a-10ad-4a2b-b4d4-e682ae7f4a99_isrForCrosstalkSources_946687_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/9f5dbf9a-10ad-4a2b-b4d4-e682ae7f4a99_isrForCrosstalkSources_946687_9.stderr
parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e4e1353c0b686029b5776047e31ad28 with result from task 17
parsl.dataflow.dflow DEBUG: Task 17 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/3d10df6f-f4b2-42f3-a84e-dfbb8ac49a87_isrForCrosstalkSources_946688_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/3d10df6f-f4b2-42f3-a84e-dfbb8ac49a87_isrForCrosstalkSources_946688_10.stderr
parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c4d9f0d200c1e73c7f8257dd312facdd with result from task 10
parsl.dataflow.dflow DEBUG: Task 10 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b7e75e57-710d-4e1f-9ee7-22502b76ecfb_isrForCrosstalkSources_946688_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b7e75e57-710d-4e1f-9ee7-22502b76ecfb_isrForCrosstalkSources_946688_2.stderr
parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e0147e7732c885e2d65530cbc20a154 with result from task 6
parsl.dataflow.dflow DEBUG: Task 6 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/39fc9517-fd22-4130-884f-e0611838c68b_isrForCrosstalkSources_946688_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/39fc9517-fd22-4130-884f-e0611838c68b_isrForCrosstalkSources_946688_7.stderr
parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b9c15fbdb23daa132468a8348ae578f with result from task 16
parsl.dataflow.dflow DEBUG: Task 16 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/22494117-4be5-40ef-98d9-8d3da433eeac_isrForCrosstalkSources_946688_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/22494117-4be5-40ef-98d9-8d3da433eeac_isrForCrosstalkSources_946688_3.stderr
parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8862012b117e9e90bd01604ab8cda678 with result from task 5
parsl.dataflow.dflow DEBUG: Task 5 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f6272109-9a1a-4404-8865-90e4baa663b1_isrForCrosstalkSources_946686_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f6272109-9a1a-4404-8865-90e4baa663b1_isrForCrosstalkSources_946686_1.stderr
parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 183cbd48bb669b10e3b001b199a433a5 with result from task 15
parsl.dataflow.dflow DEBUG: Task 15 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/319648de-ef82-46fb-bf8d-f0f47c8cde45_isrForCrosstalkSources_946686_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/319648de-ef82-46fb-bf8d-f0f47c8cde45_isrForCrosstalkSources_946686_38.stderr
parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e7a9d0011f83abfe6fbc2ea233245b5b with result from task 13
parsl.dataflow.dflow DEBUG: Task 13 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/926e74a7-012d-498a-8404-8dfe6ae5ebaf_isrForCrosstalkSources_946688_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/926e74a7-012d-498a-8404-8dfe6ae5ebaf_isrForCrosstalkSources_946688_36.stderr
parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 70e8580c231f5aaa67b91675c92e7646 with result from task 9
parsl.dataflow.dflow DEBUG: Task 9 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/8eb91ebc-eeac-42d8-bb32-ad68682b853a_isrForCrosstalkSources_946686_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/8eb91ebc-eeac-42d8-bb32-ad68682b853a_isrForCrosstalkSources_946686_54.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 162
parsl.dataflow.strategy DEBUG: Executor multi has 162 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2e46ba575311041c0b6616209771840c with result from task 19
parsl.dataflow.dflow DEBUG: Task 19 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/177153dc-0c37-4719-beb1-c75330f8d763_isrForCrosstalkSources_946687_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/177153dc-0c37-4719-beb1-c75330f8d763_isrForCrosstalkSources_946687_26.stderr
parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6cf9aed0eea8e3a7ee277513bc7c09a1 with result from task 20
parsl.dataflow.dflow DEBUG: Task 20 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fd29e63f-5a91-4207-b028-7eba491e6dbb_isrForCrosstalkSources_946687_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fd29e63f-5a91-4207-b028-7eba491e6dbb_isrForCrosstalkSources_946687_37.stderr
parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43013db986104468288d30d138202702 with result from task 23
parsl.dataflow.dflow DEBUG: Task 23 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/dfa3be0f-23fe-48aa-8925-5130ff0774b5_isrForCrosstalkSources_946687_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/dfa3be0f-23fe-48aa-8925-5130ff0774b5_isrForCrosstalkSources_946687_53.stderr
parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e1b21fdec2516dc6da4f029196dc708f with result from task 24
parsl.dataflow.dflow DEBUG: Task 24 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/beebdcec-3409-449e-bf04-933d5114b31c_isrForCrosstalkSources_946687_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/beebdcec-3409-449e-bf04-933d5114b31c_isrForCrosstalkSources_946687_60.stderr
parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c542096f043a58edf936e5e4732739ff with result from task 18
parsl.dataflow.dflow DEBUG: Task 18 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/d3c991f1-b6c5-4a01-be1b-512435fc2ff2_isrForCrosstalkSources_946687_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/d3c991f1-b6c5-4a01-be1b-512435fc2ff2_isrForCrosstalkSources_946687_15.stderr
parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 208742476e792804d310701dafd11d44 with result from task 21
parsl.dataflow.dflow DEBUG: Task 21 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ca37e732-57c2-4920-9180-614f4a12fb08_isrForCrosstalkSources_946686_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ca37e732-57c2-4920-9180-614f4a12fb08_isrForCrosstalkSources_946686_17.stderr
parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2f319d5a039bca0eab7803b96e5dc278 with result from task 22
parsl.dataflow.dflow DEBUG: Task 22 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ba4ca7ff-615e-4a7b-9c9a-bc74dd235835_isrForCrosstalkSources_946687_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ba4ca7ff-615e-4a7b-9c9a-bc74dd235835_isrForCrosstalkSources_946687_61.stderr
parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36480f15708c6dfe308fedcd3d454fa4 with result from task 58
parsl.dataflow.dflow DEBUG: Task 58 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/bbc705c2-da61-47fc-8b7d-8cb998f3e08c_isrForCrosstalkSources_946686_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/bbc705c2-da61-47fc-8b7d-8cb998f3e08c_isrForCrosstalkSources_946686_9.stderr
parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3ab559416f61c59be25a4abd9f44bac7 with result from task 59
parsl.dataflow.dflow DEBUG: Task 59 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/7e3ff471-f68b-4659-ac98-1cae70a0ebb3_isrForCrosstalkSources_946688_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/7e3ff471-f68b-4659-ac98-1cae70a0ebb3_isrForCrosstalkSources_946688_41.stderr
parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc32393f8b7047ff0acd0e2961b7570 with result from task 31
parsl.dataflow.dflow DEBUG: Task 31 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8af63715-54d1-4c8e-b73e-dd4a07f147c8_isrForCrosstalkSources_946688_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8af63715-54d1-4c8e-b73e-dd4a07f147c8_isrForCrosstalkSources_946688_24.stderr
parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab1081c46f41247c72e4428f1472dc1b with result from task 37
parsl.dataflow.dflow DEBUG: Task 37 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f4286ed7-f4d9-4244-93fb-c9413ce9b205_isrForCrosstalkSources_946687_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f4286ed7-f4d9-4244-93fb-c9413ce9b205_isrForCrosstalkSources_946687_51.stderr
parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf85724ed16a0e01a09f47ea690d134f with result from task 30
parsl.dataflow.dflow DEBUG: Task 30 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/48f5a42a-9fe1-4aa5-838a-9fcdb4a2c91a_isrForCrosstalkSources_946688_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/48f5a42a-9fe1-4aa5-838a-9fcdb4a2c91a_isrForCrosstalkSources_946688_13.stderr
parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c2bd29eaa1e54e9da0b9867a260236ec with result from task 34
parsl.dataflow.dflow DEBUG: Task 34 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/5b60a9f9-e31d-45e1-913a-21e9b83d5740_isrForCrosstalkSources_946687_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/5b60a9f9-e31d-45e1-913a-21e9b83d5740_isrForCrosstalkSources_946687_11.stderr
parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d8af0adec26d1de835a7158e029831e with result from task 32
parsl.dataflow.dflow DEBUG: Task 32 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e8a1154a-04c6-4238-acc0-e14d18f88661_isrForCrosstalkSources_946687_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e8a1154a-04c6-4238-acc0-e14d18f88661_isrForCrosstalkSources_946687_30.stderr
parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 850b7e02262e39c8cc97f58275d76751 with result from task 27
parsl.dataflow.dflow DEBUG: Task 27 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/f5eb4652-71e0-4ebb-9744-df7ef63c4234_isrForCrosstalkSources_946688_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/f5eb4652-71e0-4ebb-9744-df7ef63c4234_isrForCrosstalkSources_946688_12.stderr
parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa8c60b99dcd40cc3c6771e9476c6e31 with result from task 28
parsl.dataflow.dflow DEBUG: Task 28 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/186a47a4-85d4-4837-aa65-631a1cf44891_isrForCrosstalkSources_946687_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/186a47a4-85d4-4837-aa65-631a1cf44891_isrForCrosstalkSources_946687_54.stderr
parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b86581f6f97416baf4664d64d7b6488a with result from task 33
parsl.dataflow.dflow DEBUG: Task 33 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/bfad661f-c0df-4b3c-a796-6d128d40bc4c_isrForCrosstalkSources_946688_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/bfad661f-c0df-4b3c-a796-6d128d40bc4c_isrForCrosstalkSources_946688_8.stderr
parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24b5d41bb52e2f3736076e6c3581097c with result from task 29
parsl.dataflow.dflow DEBUG: Task 29 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/b52aff84-6573-4348-b68b-ccb5386e83e2_isrForCrosstalkSources_946687_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/b52aff84-6573-4348-b68b-ccb5386e83e2_isrForCrosstalkSources_946687_19.stderr
parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62a0ae01823f2b87da7f00f4044be5aa with result from task 36
parsl.dataflow.dflow DEBUG: Task 36 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/247ad116-a7b6-4e54-9215-038b9d62f3b8_isrForCrosstalkSources_946686_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/247ad116-a7b6-4e54-9215-038b9d62f3b8_isrForCrosstalkSources_946686_22.stderr
parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 843b1524209fe2f2671ec390cdd9a403 with result from task 35
parsl.dataflow.dflow DEBUG: Task 35 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cd70a8c6-c39b-483a-a7a6-30610879565c_isrForCrosstalkSources_946686_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cd70a8c6-c39b-483a-a7a6-30610879565c_isrForCrosstalkSources_946686_60.stderr
2024-03-14 02:52:24 proc_lsst.multi:146 [INFO]  found job 13859 in provider local
proc_lsst.multi INFO: found job 13859 in provider local
2024-03-14 02:52:24 proc_lsst.multi:146 [INFO]  found job 17028877 in provider astro
proc_lsst.multi INFO: found job 17028877 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:24 proc_lsst.multi:146 [INFO]  found job 17028878 in provider astro
proc_lsst.multi INFO: found job 17028878 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:25 proc_lsst.multi:146 [INFO]  found job 17028879 in provider astro
proc_lsst.multi INFO: found job 17028879 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aefe7fadf828f39ebfa696575758c0ff with result from task 62
parsl.dataflow.dflow DEBUG: Task 62 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4d1617dc-4a0f-4418-9562-f27d06b15fcb_isrForCrosstalkSources_946687_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4d1617dc-4a0f-4418-9562-f27d06b15fcb_isrForCrosstalkSources_946687_23.stderr
parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 608a050d1ed836f8128fa0fdc303bb8f with result from task 67
parsl.dataflow.dflow DEBUG: Task 67 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/23f2166a-52fb-4458-b731-4f5ed588fbdd_isrForCrosstalkSources_946687_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/23f2166a-52fb-4458-b731-4f5ed588fbdd_isrForCrosstalkSources_946687_17.stderr
parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab1b3766390edaece7784d25d86cf338 with result from task 68
parsl.dataflow.dflow DEBUG: Task 68 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f55311a3-52d5-41e0-ac2d-43b2f5b40f68_isrForCrosstalkSources_946686_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f55311a3-52d5-41e0-ac2d-43b2f5b40f68_isrForCrosstalkSources_946686_27.stderr
parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7047f84bc93be00db117e8b33cf1d207 with result from task 65
parsl.dataflow.dflow DEBUG: Task 65 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/7d4d37a3-3631-448e-886f-be689d212c16_isrForCrosstalkSources_946686_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/7d4d37a3-3631-448e-886f-be689d212c16_isrForCrosstalkSources_946686_28.stderr
parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b31d1fec6d2e804c3b2da0abe2ba119 with result from task 64
parsl.dataflow.dflow DEBUG: Task 64 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2d3c9e1d-81a1-415f-9960-2f2be0412ba0_isrForCrosstalkSources_946688_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2d3c9e1d-81a1-415f-9960-2f2be0412ba0_isrForCrosstalkSources_946688_15.stderr
parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa26d2e9808b6d971bbe0228d0ff291 with result from task 60
parsl.dataflow.dflow DEBUG: Task 60 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e4d628af-dab3-41b9-b451-d27dfa8f629b_isrForCrosstalkSources_946686_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e4d628af-dab3-41b9-b451-d27dfa8f629b_isrForCrosstalkSources_946686_51.stderr
parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba53bc12a745a2dbfbde9a7c16a5c0ba with result from task 63
parsl.dataflow.dflow DEBUG: Task 63 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/50f54a4a-708f-4ec3-bccf-8508f748ce6d_isrForCrosstalkSources_946686_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/50f54a4a-708f-4ec3-bccf-8508f748ce6d_isrForCrosstalkSources_946686_58.stderr
parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d526bd0eed523260fb761371351d01a5 with result from task 66
parsl.dataflow.dflow DEBUG: Task 66 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2957f4e7-f6d1-4a05-8c7f-615713fcbe74_isrForCrosstalkSources_946688_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2957f4e7-f6d1-4a05-8c7f-615713fcbe74_isrForCrosstalkSources_946688_25.stderr
parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62259e181d6407c2ac36024c41946cf8 with result from task 71
parsl.dataflow.dflow DEBUG: Task 71 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/3ca88fdf-15cf-416c-8dc4-a5b807653343_isrForCrosstalkSources_946687_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/3ca88fdf-15cf-416c-8dc4-a5b807653343_isrForCrosstalkSources_946687_5.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:25 proc_lsst.multi:146 [INFO]  found job 17028880 in provider astro
proc_lsst.multi INFO: found job 17028880 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a2c5ab2535fc46bcac297eca67aaeec with result from task 70
parsl.dataflow.dflow DEBUG: Task 70 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/203a81ef-1bc0-4225-9f99-9aef2301577a_isrForCrosstalkSources_946688_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/203a81ef-1bc0-4225-9f99-9aef2301577a_isrForCrosstalkSources_946688_31.stderr
parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b77ea52b2cb2c144a20668876ac9dca with result from task 61
parsl.dataflow.dflow DEBUG: Task 61 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/56bb7a21-eb14-4eab-b776-2349adde6b2f_isrForCrosstalkSources_946687_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/56bb7a21-eb14-4eab-b776-2349adde6b2f_isrForCrosstalkSources_946687_41.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:25 proc_lsst.multi:146 [INFO]  found job 17028881 in provider astro
proc_lsst.multi INFO: found job 17028881 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6d9ab095c2f396844ac98fb622ce2da3 with result from task 69
parsl.dataflow.dflow DEBUG: Task 69 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/476177f6-3b8c-4b6d-bbf2-9e51fdd5972f_isrForCrosstalkSources_946687_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/476177f6-3b8c-4b6d-bbf2-9e51fdd5972f_isrForCrosstalkSources_946687_62.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:25 proc_lsst.multi:146 [INFO]  found job 17028882 in provider astro
proc_lsst.multi INFO: found job 17028882 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:26 proc_lsst.multi:146 [INFO]  found job 17028883 in provider astro
proc_lsst.multi INFO: found job 17028883 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:26 proc_lsst.multi:146 [INFO]  found job 17028884 in provider astro
proc_lsst.multi INFO: found job 17028884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:26 proc_lsst.multi:146 [INFO]  found job 17028885 in provider astro
proc_lsst.multi INFO: found job 17028885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:26 proc_lsst.multi:146 [INFO]  found job 17028886 in provider astro
proc_lsst.multi INFO: found job 17028886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:27 proc_lsst.multi:146 [INFO]  found job 17028887 in provider astro
proc_lsst.multi INFO: found job 17028887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0613623d760411f3162d311afe868c8a with result from task 79
parsl.dataflow.dflow DEBUG: Task 79 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c9fb1fa7-bc54-4bc5-83cc-e8e0f33ea670_isrForCrosstalkSources_946688_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c9fb1fa7-bc54-4bc5-83cc-e8e0f33ea670_isrForCrosstalkSources_946688_32.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:27 proc_lsst.multi:146 [INFO]  found job 17028888 in provider astro
proc_lsst.multi INFO: found job 17028888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:27 proc_lsst.multi:146 [INFO]  found job 17028889 in provider astro
proc_lsst.multi INFO: found job 17028889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88eefd379f7d21e8b82b32fe3c993621 with result from task 39
parsl.dataflow.dflow DEBUG: Task 39 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/3f4b60f6-7913-45ce-87b3-ab1ba543d9e2_isrForCrosstalkSources_946686_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/3f4b60f6-7913-45ce-87b3-ab1ba543d9e2_isrForCrosstalkSources_946686_34.stderr
parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7afef2ca224e2da7d325c1973e130c39 with result from task 42
parsl.dataflow.dflow DEBUG: Task 42 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/291e3683-916f-4009-aa27-fcf12cb0ae0f_isrForCrosstalkSources_946686_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/291e3683-916f-4009-aa27-fcf12cb0ae0f_isrForCrosstalkSources_946686_57.stderr
parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 83dd2c2666c590043b963745c728064c with result from task 38
parsl.dataflow.dflow DEBUG: Task 38 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5aa21ed4-1bc6-46ef-a416-a59145d0a8d4_isrForCrosstalkSources_946688_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5aa21ed4-1bc6-46ef-a416-a59145d0a8d4_isrForCrosstalkSources_946688_48.stderr
parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 37e2ca372ff09923b408635d4bd27bfd with result from task 41
parsl.dataflow.dflow DEBUG: Task 41 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/4cf75b3f-57f7-4916-961b-5deaa43327a2_isrForCrosstalkSources_946688_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/4cf75b3f-57f7-4916-961b-5deaa43327a2_isrForCrosstalkSources_946688_44.stderr
parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 407a2b61a6f800209b6b9133fdf19a17 with result from task 40
parsl.dataflow.dflow DEBUG: Task 40 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/88f86847-c867-4882-bd71-159dd2fdc431_isrForCrosstalkSources_946687_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/88f86847-c867-4882-bd71-159dd2fdc431_isrForCrosstalkSources_946687_42.stderr
parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 435811480da25987a9b0c34491a28cd7 with result from task 47
parsl.dataflow.dflow DEBUG: Task 47 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/55be47dc-55e4-404c-836e-61e2973f7c08_isrForCrosstalkSources_946686_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/55be47dc-55e4-404c-836e-61e2973f7c08_isrForCrosstalkSources_946686_18.stderr
parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4cecce01e7e46a224556d5180c2b18b5 with result from task 48
parsl.dataflow.dflow DEBUG: Task 48 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2c5611b7-52e7-4fd7-9d12-7f6680a677bb_isrForCrosstalkSources_946688_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2c5611b7-52e7-4fd7-9d12-7f6680a677bb_isrForCrosstalkSources_946688_54.stderr
parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 69c15957b5be9f3f14cb47124f74ae73 with result from task 45
parsl.dataflow.dflow DEBUG: Task 45 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/bffca539-9f82-49d4-96d5-5b38232bb1a3_isrForCrosstalkSources_946688_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/bffca539-9f82-49d4-96d5-5b38232bb1a3_isrForCrosstalkSources_946688_11.stderr
parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b023b2ddd737d1ce5a213e751dc7a8f4 with result from task 50
parsl.dataflow.dflow DEBUG: Task 50 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/6cf5f3a0-8d61-4d77-a12c-324ac74942ab_isrForCrosstalkSources_946687_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/6cf5f3a0-8d61-4d77-a12c-324ac74942ab_isrForCrosstalkSources_946687_56.stderr
parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c6b8f083a51bb5769cf4556e1cbe660f with result from task 49
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:27 proc_lsst.multi:146 [INFO]  found job 17028890 in provider astro
proc_lsst.multi INFO: found job 17028890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow DEBUG: Task 49 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/39b24d0c-2282-4d90-b71e-effeb9815fef_isrForCrosstalkSources_946688_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/39b24d0c-2282-4d90-b71e-effeb9815fef_isrForCrosstalkSources_946688_60.stderr
parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 86cd7b28de2cc6f9523da881efe19ad3 with result from task 53
parsl.dataflow.dflow DEBUG: Task 53 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/57eb1bc3-ddef-4ba9-87bc-b07c5db00b2e_isrForCrosstalkSources_946686_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/57eb1bc3-ddef-4ba9-87bc-b07c5db00b2e_isrForCrosstalkSources_946686_45.stderr
parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d1777a423f119ceced9c667942a77f03 with result from task 43
parsl.dataflow.dflow DEBUG: Task 43 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2059dee1-3638-4fc7-b3d6-679b2650d02a_isrForCrosstalkSources_946688_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/2059dee1-3638-4fc7-b3d6-679b2650d02a_isrForCrosstalkSources_946688_1.stderr
parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a80b16887b4d0ef603aa8a78b99b013 with result from task 46
parsl.dataflow.dflow DEBUG: Task 46 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/d65d5469-3472-4a5d-be0a-e493f5f2815b_isrForCrosstalkSources_946688_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/d65d5469-3472-4a5d-be0a-e493f5f2815b_isrForCrosstalkSources_946688_46.stderr
parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4eafc9010df716e84afe9116516beafa with result from task 52
parsl.dataflow.dflow DEBUG: Task 52 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/740dd3d5-b9e5-4c60-9b09-e6c5d862ef67_isrForCrosstalkSources_946687_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/740dd3d5-b9e5-4c60-9b09-e6c5d862ef67_isrForCrosstalkSources_946687_52.stderr
parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f873a87a26e8a5a99580d026e7a14d8e with result from task 51
parsl.dataflow.dflow DEBUG: Task 51 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d9396dd8-3ce9-4eaa-b5a2-6b8db69348fb_isrForCrosstalkSources_946686_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d9396dd8-3ce9-4eaa-b5a2-6b8db69348fb_isrForCrosstalkSources_946686_39.stderr
parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 163a64f2d21196c285ad883e32f284ba with result from task 44
parsl.dataflow.dflow DEBUG: Task 44 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/90f67fcb-d50d-47ed-b0ab-ff165b6492e5_isrForCrosstalkSources_946688_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/90f67fcb-d50d-47ed-b0ab-ff165b6492e5_isrForCrosstalkSources_946688_61.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:28 proc_lsst.multi:146 [INFO]  found job 17028891 in provider astro
proc_lsst.multi INFO: found job 17028891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:28 proc_lsst.multi:146 [INFO]  found job 17028892 in provider astro
proc_lsst.multi INFO: found job 17028892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:28 proc_lsst.multi:146 [INFO]  found job 17028893 in provider astro
proc_lsst.multi INFO: found job 17028893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:28 proc_lsst.multi:146 [INFO]  found job 17028894 in provider astro
proc_lsst.multi INFO: found job 17028894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8bbbe6a768c52132687eb0b1541032a with result from task 80
parsl.dataflow.dflow DEBUG: Task 80 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/24a1ab8d-5f6b-4337-8f30-53a649fd550a_isrForCrosstalkSources_946686_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/24a1ab8d-5f6b-4337-8f30-53a649fd550a_isrForCrosstalkSources_946686_53.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:28 proc_lsst.multi:146 [INFO]  found job 17028895 in provider astro
proc_lsst.multi INFO: found job 17028895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:29 proc_lsst.multi:146 [INFO]  found job 17028896 in provider astro
proc_lsst.multi INFO: found job 17028896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:29 proc_lsst.multi:146 [INFO]  found job 17028897 in provider astro
proc_lsst.multi INFO: found job 17028897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:29 proc_lsst.multi:146 [INFO]  found job 17028898 in provider astro
proc_lsst.multi INFO: found job 17028898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:30 proc_lsst.multi:146 [INFO]  found job 17028899 in provider astro
proc_lsst.multi INFO: found job 17028899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8ece0ee3159cb6612b959ef23f68a187 with result from task 75
parsl.dataflow.dflow DEBUG: Task 75 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/12923e8b-424b-4ba0-8648-5abacd9549ae_isrForCrosstalkSources_946686_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/12923e8b-424b-4ba0-8648-5abacd9549ae_isrForCrosstalkSources_946686_26.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:30 proc_lsst.multi:146 [INFO]  found job 17028900 in provider astro
proc_lsst.multi INFO: found job 17028900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a1cf5f8a489e00f88a81ad89a10da3e with result from task 74
parsl.dataflow.dflow DEBUG: Task 74 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/60361637-f405-478b-bdc1-e231901a3644_isrForCrosstalkSources_946688_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/60361637-f405-478b-bdc1-e231901a3644_isrForCrosstalkSources_946688_19.stderr
parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5a22a9c56e32d0986ee23dabd799b70 with result from task 76
parsl.dataflow.dflow DEBUG: Task 76 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8f827929-5de1-43d6-ade4-73b8985b1de1_isrForCrosstalkSources_946688_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8f827929-5de1-43d6-ade4-73b8985b1de1_isrForCrosstalkSources_946688_39.stderr
parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e4cc3a9c68da3e1a26201bd9ce83a8c with result from task 73
parsl.dataflow.dflow DEBUG: Task 73 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f18c156f-b97a-44ba-b91c-b28554badaa6_isrForCrosstalkSources_946686_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/f18c156f-b97a-44ba-b91c-b28554badaa6_isrForCrosstalkSources_946686_12.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:30 proc_lsst.multi:146 [INFO]  found job 17028901 in provider astro
proc_lsst.multi INFO: found job 17028901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8305f7e2e9031d16724858f11164914 with result from task 77
parsl.dataflow.dflow DEBUG: Task 77 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/1899f4b0-d615-4eff-af8e-182f005c52bb_isrForCrosstalkSources_946687_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/1899f4b0-d615-4eff-af8e-182f005c52bb_isrForCrosstalkSources_946687_50.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:31 proc_lsst.multi:146 [INFO]  found job 17028902 in provider astro
proc_lsst.multi INFO: found job 17028902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91172527ee9a218deed63cbc3fe72594 with result from task 78
parsl.dataflow.dflow DEBUG: Task 78 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/77290212-df59-40a4-8337-d2084f8d0981_isrForCrosstalkSources_946686_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/77290212-df59-40a4-8337-d2084f8d0981_isrForCrosstalkSources_946686_42.stderr
parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 62e060279a86e1bd7c019f30b8f54989 with result from task 72
parsl.dataflow.dflow DEBUG: Task 72 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/7a3663cd-5514-4196-a47b-714d003007e3_isrForCrosstalkSources_946688_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/7a3663cd-5514-4196-a47b-714d003007e3_isrForCrosstalkSources_946688_58.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:31 proc_lsst.multi:146 [INFO]  found job 17028903 in provider astro
proc_lsst.multi INFO: found job 17028903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:31 proc_lsst.multi:146 [INFO]  found job 17028904 in provider astro
proc_lsst.multi INFO: found job 17028904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:32 proc_lsst.multi:146 [INFO]  found job 17028905 in provider astro
proc_lsst.multi INFO: found job 17028905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:32 proc_lsst.multi:146 [INFO]  found job 17028906 in provider astro
proc_lsst.multi INFO: found job 17028906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:32 proc_lsst.multi:146 [INFO]  found job 17028908 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028908 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:32 proc_lsst.multi:146 [INFO]  found job 17028909 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028909 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:32 proc_lsst.multi:146 [INFO]  found job 17028910 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028910 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:33 proc_lsst.multi:146 [INFO]  found job 17028911 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028911 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:33 proc_lsst.multi:146 [INFO]  found job 17028912 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028912 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:33 proc_lsst.multi:146 [INFO]  found job 17028913 in provider escience_a40
proc_lsst.multi INFO: found job 17028913 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:33 proc_lsst.multi:146 [INFO]  found job 17028914 in provider escience_a40
proc_lsst.multi INFO: found job 17028914 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:34 proc_lsst.multi:146 [INFO]  found job 17028915 in provider escience_a40
proc_lsst.multi INFO: found job 17028915 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:34 proc_lsst.multi:146 [INFO]  found job 17028916 in provider escience_a40
proc_lsst.multi INFO: found job 17028916 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd6a283af6ab13c28e8ab75a69a1e95c with result from task 84
parsl.dataflow.dflow DEBUG: Task 84 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/df0c8c75-25f6-4466-92e1-5082d62fd173_isrForCrosstalkSources_946688_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/df0c8c75-25f6-4466-92e1-5082d62fd173_isrForCrosstalkSources_946688_14.stderr
parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c1b28d195b29168594a8d8d4b2dd2e9d with result from task 81
parsl.dataflow.dflow DEBUG: Task 81 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/45861880-6309-45d8-82e7-ca2890a513fc_isrForCrosstalkSources_946688_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/45861880-6309-45d8-82e7-ca2890a513fc_isrForCrosstalkSources_946688_5.stderr
parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7f830b8784850ca158498c138bfa3feb with result from task 87
parsl.dataflow.dflow DEBUG: Task 87 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f897d711-d7b3-4acd-9ff7-8b0d1336e43a_isrForCrosstalkSources_946687_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/f897d711-d7b3-4acd-9ff7-8b0d1336e43a_isrForCrosstalkSources_946687_2.stderr
parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5bc93e9307f26c27e393e4107389f79b with result from task 88
parsl.dataflow.dflow DEBUG: Task 88 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/909239b8-a4fb-43b7-9656-6e0083b8210c_isrForCrosstalkSources_946687_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/909239b8-a4fb-43b7-9656-6e0083b8210c_isrForCrosstalkSources_946687_31.stderr
parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 76d797a4a7f94673b778f254edf8c7dd with result from task 85
parsl.dataflow.dflow DEBUG: Task 85 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/623742d7-02bb-4907-bcdc-107e94956c8b_isrForCrosstalkSources_946687_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/623742d7-02bb-4907-bcdc-107e94956c8b_isrForCrosstalkSources_946687_12.stderr
parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52b86a30d7f4bef9c7dd2c9a9f451ed4 with result from task 86
parsl.dataflow.dflow DEBUG: Task 86 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/5db29991-e888-426d-9116-5f116c845b60_isrForCrosstalkSources_946687_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/5db29991-e888-426d-9116-5f116c845b60_isrForCrosstalkSources_946687_28.stderr
parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5796357cf552a45506af5f69cc8b26da with result from task 83
parsl.dataflow.dflow DEBUG: Task 83 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4632149f-5b29-4530-a42b-772a3e1bcf46_isrForCrosstalkSources_946686_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4632149f-5b29-4530-a42b-772a3e1bcf46_isrForCrosstalkSources_946686_4.stderr
parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8578be6dcd3ff73cf97ccfe3104640a with result from task 89
parsl.dataflow.dflow DEBUG: Task 89 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/1a9e623a-0349-4aa8-a343-5ffcc7121ba2_isrForCrosstalkSources_946686_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/1a9e623a-0349-4aa8-a343-5ffcc7121ba2_isrForCrosstalkSources_946686_14.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:34 proc_lsst.multi:146 [INFO]  found job 17028917 in provider escience_a40
proc_lsst.multi INFO: found job 17028917 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4586d918ebcc25a8da3096828ae390fb with result from task 91
parsl.dataflow.dflow DEBUG: Task 91 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/12a605c1-f712-4e99-8235-39e00dd3bc22_isrForCrosstalkSources_946688_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/12a605c1-f712-4e99-8235-39e00dd3bc22_isrForCrosstalkSources_946688_16.stderr
parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35b475516d2c1e0476196208db0cc1a5 with result from task 90
parsl.dataflow.dflow DEBUG: Task 90 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e2fa4a26-bf27-480c-a09a-72d14a1a2e52_isrForCrosstalkSources_946687_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e2fa4a26-bf27-480c-a09a-72d14a1a2e52_isrForCrosstalkSources_946687_7.stderr
parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ea984c5f0065a537d75db287dbb9dd23 with result from task 82
parsl.dataflow.dflow DEBUG: Task 82 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/1cb3f4d5-441d-43a1-b8a7-3ba214b6e9c4_isrForCrosstalkSources_946686_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/1cb3f4d5-441d-43a1-b8a7-3ba214b6e9c4_isrForCrosstalkSources_946686_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:35 proc_lsst.multi:146 [INFO]  found job 17028918 in provider escience_a40
proc_lsst.multi INFO: found job 17028918 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a8af4f037570f497155da1a553ada4c5 with result from task 102
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:35 proc_lsst.multi:146 [INFO]  found job 17028919 in provider escience_a40
proc_lsst.multi INFO: found job 17028919 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.dataflow.dflow DEBUG: Task 102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8ab96b33-6a33-40c1-80fc-08943489bee7_isrForCrosstalkSources_946688_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8ab96b33-6a33-40c1-80fc-08943489bee7_isrForCrosstalkSources_946688_20.stderr
parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c227fa17606aaeffadb6f230efdb58e with result from task 93
parsl.dataflow.dflow DEBUG: Task 93 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/3a639762-854b-4fc1-9858-9d17f237d0e0_isrForCrosstalkSources_946687_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/3a639762-854b-4fc1-9858-9d17f237d0e0_isrForCrosstalkSources_946687_4.stderr
parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8d254dbf4881fb11be0e817249213dae with result from task 100
parsl.dataflow.dflow DEBUG: Task 100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e5f1bc64-6320-4a06-b9fe-60057b164ea1_isrForCrosstalkSources_946687_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/e5f1bc64-6320-4a06-b9fe-60057b164ea1_isrForCrosstalkSources_946687_1.stderr
parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 74e2954c3770b2aea6e362c314903e14 with result from task 103
parsl.dataflow.dflow DEBUG: Task 103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/300bd0c3-f1d7-4c4d-a30f-c8786d78e2a0_isrForCrosstalkSources_946687_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/300bd0c3-f1d7-4c4d-a30f-c8786d78e2a0_isrForCrosstalkSources_946687_47.stderr
parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ea105c1b42722569de53ba4ec0634aa3 with result from task 101
parsl.dataflow.dflow DEBUG: Task 101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/035fca6d-148a-4ffc-87a6-ca3f7323d72c_isrForCrosstalkSources_946687_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/035fca6d-148a-4ffc-87a6-ca3f7323d72c_isrForCrosstalkSources_946687_36.stderr
parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d88347f351310b97f1d7783a8caec828 with result from task 96
parsl.dataflow.dflow DEBUG: Task 96 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/498d9138-8790-494a-b340-fbebe1a31499_isrForCrosstalkSources_946686_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/498d9138-8790-494a-b340-fbebe1a31499_isrForCrosstalkSources_946686_5.stderr
parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9b3aa243f2d0c3035fc4eac94c24485 with result from task 97
parsl.dataflow.dflow DEBUG: Task 97 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/5714a688-6733-407e-b2d2-b43056b14af9_isrForCrosstalkSources_946686_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/5714a688-6733-407e-b2d2-b43056b14af9_isrForCrosstalkSources_946686_21.stderr
parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4b661b7e4ca115996f7f588f7f7f5ef0 with result from task 92
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:35 proc_lsst.multi:146 [INFO]  found job 17028920 in provider escience_a40
proc_lsst.multi INFO: found job 17028920 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.dataflow.dflow DEBUG: Task 92 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ed036f12-b8f5-4d0a-9bc3-2ede82268269_isrForCrosstalkSources_946686_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ed036f12-b8f5-4d0a-9bc3-2ede82268269_isrForCrosstalkSources_946686_40.stderr
parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a18aae9d33be85090628738faab021e with result from task 99
parsl.dataflow.dflow DEBUG: Task 99 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/38d88613-3dec-4a7b-ba83-6ee823ed4523_isrForCrosstalkSources_946686_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/38d88613-3dec-4a7b-ba83-6ee823ed4523_isrForCrosstalkSources_946686_31.stderr
parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 813938d29b40f4514e309b32b8bbc032 with result from task 98
parsl.dataflow.dflow DEBUG: Task 98 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/9eb6fcc8-3468-4c7e-982c-b5798d9776eb_isrForCrosstalkSources_946688_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/9eb6fcc8-3468-4c7e-982c-b5798d9776eb_isrForCrosstalkSources_946688_55.stderr
parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 04bd669faa15694914e89d250da431e1 with result from task 95
parsl.dataflow.dflow DEBUG: Task 95 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e14804c3-817d-475a-ad38-b2541d15f9f1_isrForCrosstalkSources_946688_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e14804c3-817d-475a-ad38-b2541d15f9f1_isrForCrosstalkSources_946688_62.stderr
parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d0e0fa1c673b5f8203bbe375d7f7a60 with result from task 94
parsl.dataflow.dflow DEBUG: Task 94 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/6d343fd2-347f-4471-937a-c9d5579be555_isrForCrosstalkSources_946688_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/6d343fd2-347f-4471-937a-c9d5579be555_isrForCrosstalkSources_946688_59.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:36 proc_lsst.multi:146 [INFO]  found job 17028921 in provider escience_a40
proc_lsst.multi INFO: found job 17028921 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:36 proc_lsst.multi:146 [INFO]  found job 17028922 in provider escience_a40
proc_lsst.multi INFO: found job 17028922 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:36 proc_lsst.multi:146 [INFO]  found job 17028923 in provider escience_a40
proc_lsst.multi INFO: found job 17028923 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 82
parsl.dataflow.strategy DEBUG: Executor multi has 82 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 82
parsl.dataflow.strategy DEBUG: Executor multi has 82 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2363189b3fd96902761ee09ac7f086fa with result from task 104
parsl.dataflow.dflow DEBUG: Task 104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/879da37e-67f4-4f9a-b00b-a51192798824_isrForCrosstalkSources_946688_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/879da37e-67f4-4f9a-b00b-a51192798824_isrForCrosstalkSources_946688_18.stderr
parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b2798adbfeffbdfc333c0d904a7116a with result from task 121
parsl.dataflow.dflow DEBUG: Task 121 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/04f54793-46e2-4e26-b0c5-80633bba5215_isrForCrosstalkSources_946687_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/04f54793-46e2-4e26-b0c5-80633bba5215_isrForCrosstalkSources_946687_20.stderr
parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 20315949ba58223a12ac843593c4c337 with result from task 110
parsl.dataflow.dflow DEBUG: Task 110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cc6be176-2672-40fd-ad51-0403cff0a50f_isrForCrosstalkSources_946686_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cc6be176-2672-40fd-ad51-0403cff0a50f_isrForCrosstalkSources_946686_56.stderr
parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe3a77eb08b2ce468dea248783951081 with result from task 117
parsl.dataflow.dflow DEBUG: Task 117 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/76a31f60-ac0f-4c9d-8fb3-274c772c2c3b_isrForCrosstalkSources_946688_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/76a31f60-ac0f-4c9d-8fb3-274c772c2c3b_isrForCrosstalkSources_946688_34.stderr
parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99826b06e503fd9426944ce60150b53a with result from task 114
parsl.dataflow.dflow DEBUG: Task 114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/77e0dce5-ea5a-4333-828b-8ebe0c004584_isrForCrosstalkSources_946686_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/77e0dce5-ea5a-4333-828b-8ebe0c004584_isrForCrosstalkSources_946686_33.stderr
parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a9f1be4a65eacb0e16299b54bd661bd0 with result from task 116
parsl.dataflow.dflow DEBUG: Task 116 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/261055ab-7e69-4b6c-b5f1-e4c284cb69d5_isrForCrosstalkSources_946686_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/261055ab-7e69-4b6c-b5f1-e4c284cb69d5_isrForCrosstalkSources_946686_13.stderr
parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 753088b19649c85e0c025ccb45df48ca with result from task 113
parsl.dataflow.dflow DEBUG: Task 113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9110f5df-ac58-4dea-a34a-235fc6cb95c3_isrForCrosstalkSources_946686_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9110f5df-ac58-4dea-a34a-235fc6cb95c3_isrForCrosstalkSources_946686_46.stderr
parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ef41390d7e33bbbc367b5d117923cfe with result from task 118
parsl.dataflow.dflow DEBUG: Task 118 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/42e13784-64ae-488a-8361-a6104196ece2_isrForCrosstalkSources_946686_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/42e13784-64ae-488a-8361-a6104196ece2_isrForCrosstalkSources_946686_10.stderr
parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b8378a2b9597a45c8e8f64cb0d8ae9f with result from task 120
parsl.dataflow.dflow DEBUG: Task 120 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e6ad83f3-9dde-4327-913b-05c762193fd4_isrForCrosstalkSources_946686_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e6ad83f3-9dde-4327-913b-05c762193fd4_isrForCrosstalkSources_946686_2.stderr
parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24b2bd357e840f427648708a0a415594 with result from task 119
parsl.dataflow.dflow DEBUG: Task 119 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ef122ed1-5e4b-4cbc-899d-0620a1a19727_isrForCrosstalkSources_946687_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ef122ed1-5e4b-4cbc-899d-0620a1a19727_isrForCrosstalkSources_946687_21.stderr
parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 173c555091129452df7447edf44ddf2e with result from task 112
parsl.dataflow.dflow DEBUG: Task 112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5ba0e475-1de7-44a1-83aa-4d2b26ae6615_isrForCrosstalkSources_946688_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5ba0e475-1de7-44a1-83aa-4d2b26ae6615_isrForCrosstalkSources_946688_6.stderr
parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 081de032835e4b45500f895c94f92173 with result from task 111
parsl.dataflow.dflow DEBUG: Task 111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/028e5956-2dc0-4097-bfb4-855424cf7273_isrForCrosstalkSources_946687_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/028e5956-2dc0-4097-bfb4-855424cf7273_isrForCrosstalkSources_946687_49.stderr
parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89ad1dff1ee3e8aabd3818f5fbcfd752 with result from task 115
parsl.dataflow.dflow DEBUG: Task 115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/684a7899-b33c-4786-a945-3d9390da36ef_isrForCrosstalkSources_946686_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/684a7899-b33c-4786-a945-3d9390da36ef_isrForCrosstalkSources_946686_29.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 69
parsl.dataflow.strategy DEBUG: Executor multi has 69 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f85756f1b8dd849a61fba1e96caac77b with result from task 106
parsl.dataflow.dflow DEBUG: Task 106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/acf0863a-f03e-48f5-92d0-a3da4797f14d_isrForCrosstalkSources_946686_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/acf0863a-f03e-48f5-92d0-a3da4797f14d_isrForCrosstalkSources_946686_6.stderr
parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0f3474f75ef279134a7c571e5199f635 with result from task 122
parsl.dataflow.dflow DEBUG: Task 122 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4a970540-4a53-450d-8bb2-d0391790b2ce_isrForCrosstalkSources_946686_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4a970540-4a53-450d-8bb2-d0391790b2ce_isrForCrosstalkSources_946686_25.stderr
parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f5fbcadbdffec1e69c81cf65671e58a6 with result from task 123
parsl.dataflow.dflow DEBUG: Task 123 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/b5d2e7ce-476b-442e-8772-42fa001fb3ad_isrForCrosstalkSources_946686_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/b5d2e7ce-476b-442e-8772-42fa001fb3ad_isrForCrosstalkSources_946686_32.stderr
parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 404dc4d12d0fb9ac34514e75dc5489ed with result from task 124
parsl.dataflow.dflow DEBUG: Task 124 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a508b17e-df09-44c1-862b-f9b7236cef1a_isrForCrosstalkSources_946688_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a508b17e-df09-44c1-862b-f9b7236cef1a_isrForCrosstalkSources_946688_45.stderr
parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe251734458ee8ba075368c962cf9503 with result from task 128
parsl.dataflow.dflow DEBUG: Task 128 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c9aa980c-237b-49f9-9c88-47eacc2ea163_isrForCrosstalkSources_946688_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c9aa980c-237b-49f9-9c88-47eacc2ea163_isrForCrosstalkSources_946688_23.stderr
parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 92e24ccef8ddf1516c212bc0684a8fa9 with result from task 109
parsl.dataflow.dflow DEBUG: Task 109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9ba33c47-d693-4e83-b886-ad4ba28edce9_isrForCrosstalkSources_946686_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9ba33c47-d693-4e83-b886-ad4ba28edce9_isrForCrosstalkSources_946686_16.stderr
parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ca1f67aee2fe8b4fb7077f2063d8ceb2 with result from task 107
parsl.dataflow.dflow DEBUG: Task 107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/6be3df15-136f-47ab-8222-c6269bd4b3b4_isrForCrosstalkSources_946688_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/6be3df15-136f-47ab-8222-c6269bd4b3b4_isrForCrosstalkSources_946688_26.stderr
parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 333a8ae518d75b7527eaabcfad3b30e3 with result from task 108
parsl.dataflow.dflow DEBUG: Task 108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e081f529-8b12-4a11-9766-a2d820793e68_isrForCrosstalkSources_946686_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/e081f529-8b12-4a11-9766-a2d820793e68_isrForCrosstalkSources_946686_36.stderr
parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c85dd5da6bc6f00ca41a56be8e9b50d4 with result from task 152
parsl.dataflow.dflow DEBUG: Task 152 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/07679853-515e-4df7-8deb-fa4f233889e8_isrForCrosstalkSources_946687_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/07679853-515e-4df7-8deb-fa4f233889e8_isrForCrosstalkSources_946687_43.stderr
parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 043d5a3681155a16eb71e91684f0f508 with result from task 105
parsl.dataflow.dflow DEBUG: Task 105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/6cdf7f25-053b-4ff7-825a-6d6ccf4c9b4c_isrForCrosstalkSources_946686_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/6cdf7f25-053b-4ff7-825a-6d6ccf4c9b4c_isrForCrosstalkSources_946686_62.stderr
parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b56a6abf4ea80345705a2d9fbce56c2 with result from task 140
parsl.dataflow.dflow DEBUG: Task 140 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/fc674419-f3a7-4962-8187-6d495da7f427_isrForCrosstalkSources_946688_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/fc674419-f3a7-4962-8187-6d495da7f427_isrForCrosstalkSources_946688_9.stderr
parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry baf343639abe174033561ce325a71173 with result from task 147
parsl.dataflow.dflow DEBUG: Task 147 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fc14254b-952d-4e54-b4c1-cdbcdcd55eac_isrForCrosstalkSources_946687_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fc14254b-952d-4e54-b4c1-cdbcdcd55eac_isrForCrosstalkSources_946687_33.stderr
parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a3647143d3e21eb8f4a63c4fb5b0283 with result from task 153
parsl.dataflow.dflow DEBUG: Task 153 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/76aef661-2b0b-4b97-93b8-c329408dc6d1_isrForCrosstalkSources_946687_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/76aef661-2b0b-4b97-93b8-c329408dc6d1_isrForCrosstalkSources_946687_13.stderr
parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 44e31a1cb007a6718a1f0e79a54e590f with result from task 151
parsl.dataflow.dflow DEBUG: Task 151 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5856952f-49b9-4161-be95-635cae78fc7b_isrForCrosstalkSources_946688_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/5856952f-49b9-4161-be95-635cae78fc7b_isrForCrosstalkSources_946688_4.stderr
parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 11e047215bad19db0ac571b075a21de5 with result from task 150
parsl.dataflow.dflow DEBUG: Task 150 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ee017c49-940c-4023-8b86-4ce20eaba048_isrForCrosstalkSources_946687_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ee017c49-940c-4023-8b86-4ce20eaba048_isrForCrosstalkSources_946687_59.stderr
parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1b2fc8ac86364b9050e682d2db2e2ec4 with result from task 148
parsl.dataflow.dflow DEBUG: Task 148 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/624b7d24-759b-4604-a525-e4039a17d757_isrForCrosstalkSources_946688_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/624b7d24-759b-4604-a525-e4039a17d757_isrForCrosstalkSources_946688_40.stderr
parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cf8cdb4d46c5d1170d4eb27f740c6edc with result from task 144
parsl.dataflow.dflow DEBUG: Task 144 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/8c629aae-66b6-4f30-8390-613c955e99dd_isrForCrosstalkSources_946687_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/8c629aae-66b6-4f30-8390-613c955e99dd_isrForCrosstalkSources_946687_16.stderr
parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9cf034bc90f5233315d8a4c44f0402d6 with result from task 146
parsl.dataflow.dflow DEBUG: Task 146 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c86bf880-bbe9-4309-beb4-37619754a50e_isrForCrosstalkSources_946688_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/c86bf880-bbe9-4309-beb4-37619754a50e_isrForCrosstalkSources_946688_38.stderr
parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e867b2cf13082b5192f6384fa20490d6 with result from task 143
parsl.dataflow.dflow DEBUG: Task 143 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/c13543fa-6197-4a7b-a01e-682a7b6b3374_isrForCrosstalkSources_946687_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/c13543fa-6197-4a7b-a01e-682a7b6b3374_isrForCrosstalkSources_946687_29.stderr
parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b0c3f9665d9d09282e8f15ff7d327712 with result from task 145
parsl.dataflow.dflow DEBUG: Task 145 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/03bfaf7c-eae5-4491-88ce-c0cfabab36fb_isrForCrosstalkSources_946687_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/03bfaf7c-eae5-4491-88ce-c0cfabab36fb_isrForCrosstalkSources_946687_48.stderr
parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry feb0aefb9de80a23ef6c9b9f649b804d with result from task 127
parsl.dataflow.dflow DEBUG: Task 127 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/f8a0d411-b765-49cd-8ed2-332136ab5aac_isrForCrosstalkSources_946688_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/f8a0d411-b765-49cd-8ed2-332136ab5aac_isrForCrosstalkSources_946688_49.stderr
parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1f096cba13f5eabb69000835e948812e with result from task 149
parsl.dataflow.dflow DEBUG: Task 149 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/2a227480-3a94-48a8-9f4a-23b19b97944d_isrForCrosstalkSources_946686_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/2a227480-3a94-48a8-9f4a-23b19b97944d_isrForCrosstalkSources_946686_19.stderr
parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9c97354caa14178014e89e2853952725 with result from task 142
parsl.dataflow.dflow DEBUG: Task 142 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4e7da617-51ac-4aa1-a991-52e987ab445c_isrForCrosstalkSources_946686_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/4e7da617-51ac-4aa1-a991-52e987ab445c_isrForCrosstalkSources_946686_8.stderr
parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bd84cacd31b840ac04539fcf3e05916f with result from task 141
parsl.dataflow.dflow DEBUG: Task 141 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/efb61d2f-14e1-4d7f-b11a-a1183da13343_isrForCrosstalkSources_946688_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/efb61d2f-14e1-4d7f-b11a-a1183da13343_isrForCrosstalkSources_946688_42.stderr
parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9fc01054420ea7e5797a8d64596c59d8 with result from task 126
parsl.dataflow.dflow DEBUG: Task 126 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/fc12b118-5184-40d0-b7ee-44535794b891_isrForCrosstalkSources_946686_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/fc12b118-5184-40d0-b7ee-44535794b891_isrForCrosstalkSources_946686_41.stderr
parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f51c50a25565dc807fdc70afd28e67c6 with result from task 125
parsl.dataflow.dflow DEBUG: Task 125 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/68b6a7eb-f048-47bb-8c92-9fb5348ca55b_isrForCrosstalkSources_946686_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/68b6a7eb-f048-47bb-8c92-9fb5348ca55b_isrForCrosstalkSources_946686_50.stderr
parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 686202d854aea0c79c4345c0b4abb20e with result from task 130
parsl.dataflow.dflow DEBUG: Task 130 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/a7f8f13e-bf1e-48fc-93da-38cc8040c161_isrForCrosstalkSources_946687_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/a7f8f13e-bf1e-48fc-93da-38cc8040c161_isrForCrosstalkSources_946687_14.stderr
parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fba66a8e4ea872f2df79b54f2540c177 with result from task 137
parsl.dataflow.dflow DEBUG: Task 137 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/6e308271-db08-4e4c-948b-6719447ed6f8_isrForCrosstalkSources_946687_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/6e308271-db08-4e4c-948b-6719447ed6f8_isrForCrosstalkSources_946687_46.stderr
parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a756390e45168f686df402e5a4ce59d6 with result from task 131
parsl.dataflow.dflow DEBUG: Task 131 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/76002a0b-03b2-4a44-bc98-804206e3a92c_isrForCrosstalkSources_946688_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/76002a0b-03b2-4a44-bc98-804206e3a92c_isrForCrosstalkSources_946688_29.stderr
parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1116f3a459e5d15d4ca9d4b0d8bfd40d with result from task 134
parsl.dataflow.dflow DEBUG: Task 134 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/9ca6afb7-b109-4e9e-bc84-82bbecee5457_isrForCrosstalkSources_946687_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/9ca6afb7-b109-4e9e-bc84-82bbecee5457_isrForCrosstalkSources_946687_6.stderr
parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f9068958fb5e802fcf37bf9cada3a401 with result from task 129
parsl.dataflow.dflow DEBUG: Task 129 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a28e9b7f-7026-4224-bda4-5fdfd6ff1618_isrForCrosstalkSources_946688_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a28e9b7f-7026-4224-bda4-5fdfd6ff1618_isrForCrosstalkSources_946688_33.stderr
parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 05383d00f8a2df02d49660185ad1a293 with result from task 138
parsl.dataflow.dflow DEBUG: Task 138 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e0cd7943-98ef-4f0f-b1bd-2be500dd6cc4_isrForCrosstalkSources_946688_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/e0cd7943-98ef-4f0f-b1bd-2be500dd6cc4_isrForCrosstalkSources_946688_30.stderr
parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b9de51b00b4e87aac683297b28eed697 with result from task 132
parsl.dataflow.dflow DEBUG: Task 132 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ce3c2fef-3079-4119-bae3-e67f8b89126e_isrForCrosstalkSources_946687_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/ce3c2fef-3079-4119-bae3-e67f8b89126e_isrForCrosstalkSources_946687_22.stderr
parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 57a23823cc62562d830ad447f06d8ffc with result from task 139
parsl.dataflow.dflow DEBUG: Task 139 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/2347aa63-693f-4538-b98a-77fef22c75ac_isrForCrosstalkSources_946687_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/2347aa63-693f-4538-b98a-77fef22c75ac_isrForCrosstalkSources_946687_8.stderr
parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34d656abd729ffd830ad41f29ce84356 with result from task 136
parsl.dataflow.dflow DEBUG: Task 136 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/8dc284f1-d660-4099-8eea-9623102ed087_isrForCrosstalkSources_946686_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/8dc284f1-d660-4099-8eea-9623102ed087_isrForCrosstalkSources_946686_52.stderr
parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19f2c760286e31da20d6fb8aafd3f74c with result from task 135
parsl.dataflow.dflow DEBUG: Task 135 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/7caa6ccb-9495-4188-a21d-09c27a784711_isrForCrosstalkSources_946687_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/7caa6ccb-9495-4188-a21d-09c27a784711_isrForCrosstalkSources_946687_57.stderr
parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1a24348acbf0640085102a15bd0cd0dd with result from task 133
parsl.dataflow.dflow DEBUG: Task 133 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d0aee24e-778b-4464-9f1b-dae92dd20016_isrForCrosstalkSources_946686_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/d0aee24e-778b-4464-9f1b-dae92dd20016_isrForCrosstalkSources_946686_24.stderr
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 32
parsl.dataflow.strategy DEBUG: Executor multi has 32 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 32
parsl.dataflow.strategy DEBUG: Executor multi has 32 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks
parsl.dataflow.strategy DEBUG: This strategy does not support scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f947c8ac3a34c3e577b8a9aae634d497 with result from task 173
parsl.dataflow.dflow DEBUG: Task 173 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/d3d59d75-2113-4c05-8825-49dcef9e797e_isrForCrosstalkSources_946688_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/d3d59d75-2113-4c05-8825-49dcef9e797e_isrForCrosstalkSources_946688_56.stderr
parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d51086c96dbe6d25743823a69dd1a1e with result from task 177
parsl.dataflow.dflow DEBUG: Task 177 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/03149966-e2a6-4c9f-86c8-ef2562be6cea_isrForCrosstalkSources_946686_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/03149966-e2a6-4c9f-86c8-ef2562be6cea_isrForCrosstalkSources_946686_35.stderr
parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f517c9a7c68adc4687e3e3b9036e64a0 with result from task 159
parsl.dataflow.dflow DEBUG: Task 159 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/feb544b1-b9fc-48d5-8bae-204432d19eca_isrForCrosstalkSources_946687_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/feb544b1-b9fc-48d5-8bae-204432d19eca_isrForCrosstalkSources_946687_32.stderr
parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 168375e124aa1a36654af49dbbca64bc with result from task 183
parsl.dataflow.dflow DEBUG: Task 183 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b1510b04-9cb1-49b3-a80f-2054163a6c2f_isrForCrosstalkSources_946688_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/b1510b04-9cb1-49b3-a80f-2054163a6c2f_isrForCrosstalkSources_946688_51.stderr
parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f29d614fc518c68b9f00f7fcce1cac4b with result from task 175
parsl.dataflow.dflow DEBUG: Task 175 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8a50bb83-c024-4afd-b704-2a1554f721d4_isrForCrosstalkSources_946688_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8a50bb83-c024-4afd-b704-2a1554f721d4_isrForCrosstalkSources_946688_35.stderr
parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32d6c3a5d8e6a4389b90ec98c421e026 with result from task 178
parsl.dataflow.dflow DEBUG: Task 178 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/132ba0b4-6981-44ef-926e-17789d95e467_isrForCrosstalkSources_946688_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/132ba0b4-6981-44ef-926e-17789d95e467_isrForCrosstalkSources_946688_21.stderr
parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a6436f201ff75f2284a99658e0303bd with result from task 176
parsl.dataflow.dflow DEBUG: Task 176 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ddd3844e-a96e-4092-aed5-44a699f3cf19_isrForCrosstalkSources_946686_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/ddd3844e-a96e-4092-aed5-44a699f3cf19_isrForCrosstalkSources_946686_15.stderr
parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 055207fb568aba35b008e6d51b6a2722 with result from task 179
parsl.dataflow.dflow DEBUG: Task 179 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/05ab0909-541f-4b18-99e4-da5f1bd47723_isrForCrosstalkSources_946688_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/05ab0909-541f-4b18-99e4-da5f1bd47723_isrForCrosstalkSources_946688_53.stderr
parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab3aff1758655d31da9f958025035af9 with result from task 180
parsl.dataflow.dflow DEBUG: Task 180 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9a8a2113-5b7c-4070-85d1-921543b44e41_isrForCrosstalkSources_946686_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9a8a2113-5b7c-4070-85d1-921543b44e41_isrForCrosstalkSources_946686_49.stderr
parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ffe71a2cfe520cf082daed1c6c69f4c with result from task 163
parsl.dataflow.dflow DEBUG: Task 163 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8639ed1d-5a4c-4408-9709-b1e6f22f3b5b_isrForCrosstalkSources_946688_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/8639ed1d-5a4c-4408-9709-b1e6f22f3b5b_isrForCrosstalkSources_946688_28.stderr
parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3de4dc9a68cb93979424672ca06c754d with result from task 162
parsl.dataflow.dflow DEBUG: Task 162 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/23e114d7-93bd-44f6-94e9-3d22d5d2fa00_isrForCrosstalkSources_946686_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/23e114d7-93bd-44f6-94e9-3d22d5d2fa00_isrForCrosstalkSources_946686_7.stderr
parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dc80d39a258c3a38e4aefb93cd6d0ed4 with result from task 161
parsl.dataflow.dflow DEBUG: Task 161 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/c3147434-7a6f-4c0e-a883-94c166bf5f12_isrForCrosstalkSources_946686_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/c3147434-7a6f-4c0e-a883-94c166bf5f12_isrForCrosstalkSources_946686_11.stderr
parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 52e0e29e7d76106a39d93449e32fcd3c with result from task 182
parsl.dataflow.dflow DEBUG: Task 182 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/b119e2ed-110d-4301-8d42-1454bd5fc7be_isrForCrosstalkSources_946687_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/b119e2ed-110d-4301-8d42-1454bd5fc7be_isrForCrosstalkSources_946687_58.stderr
parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1ee5da1dc446cd66316c0780cd603949 with result from task 158
parsl.dataflow.dflow DEBUG: Task 158 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/9269d608-b29b-4e79-bf46-674008eb09c3_isrForCrosstalkSources_946688_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/9269d608-b29b-4e79-bf46-674008eb09c3_isrForCrosstalkSources_946688_50.stderr
parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 38682bbb9466280b8e3fd279a51453df with result from task 155
parsl.dataflow.dflow DEBUG: Task 155 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/48e595e0-3d82-4213-a216-824a0e43f103_isrForCrosstalkSources_946688_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/48e595e0-3d82-4213-a216-824a0e43f103_isrForCrosstalkSources_946688_43.stderr
parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ebdf68307ec2be7913ffa2c65ef57734 with result from task 157
parsl.dataflow.dflow DEBUG: Task 157 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/905511de-fb3a-41f6-baa8-10b6c29d6bd9_isrForCrosstalkSources_946686_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/905511de-fb3a-41f6-baa8-10b6c29d6bd9_isrForCrosstalkSources_946686_37.stderr
parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9015e25b80b7865095ec047b61375feb with result from task 185
parsl.dataflow.dflow DEBUG: Task 185 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/17ce4d14-a907-4197-8fdc-ca271b45ff3b_isrForCrosstalkSources_946688_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/17ce4d14-a907-4197-8fdc-ca271b45ff3b_isrForCrosstalkSources_946688_17.stderr
parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 26d6b29c0a1c5e1803ca25db34867149 with result from task 169
parsl.dataflow.dflow DEBUG: Task 169 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a71c4856-64ea-47a4-8ba4-2f1d3f09627e_isrForCrosstalkSources_946688_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/a71c4856-64ea-47a4-8ba4-2f1d3f09627e_isrForCrosstalkSources_946688_37.stderr
parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 630992000806ef72be18ba686f4d3311 with result from task 154
parsl.dataflow.dflow DEBUG: Task 154 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/6fd216ee-8e2d-4a57-8779-2ab64ee376d7_isrForCrosstalkSources_946686_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/6fd216ee-8e2d-4a57-8779-2ab64ee376d7_isrForCrosstalkSources_946686_61.stderr
parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fc74ce6f4ace2297f76d70535250e252 with result from task 160
parsl.dataflow.dflow DEBUG: Task 160 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/974722d3-caae-48d8-8d18-c010c322b8cc_isrForCrosstalkSources_946688_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/974722d3-caae-48d8-8d18-c010c322b8cc_isrForCrosstalkSources_946688_57.stderr
parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00c7afb7246fc99187fa3331990cb09c with result from task 156
parsl.dataflow.dflow DEBUG: Task 156 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4c1ca469-decb-4dec-906a-45ee8108ba3c_isrForCrosstalkSources_946687_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4c1ca469-decb-4dec-906a-45ee8108ba3c_isrForCrosstalkSources_946687_18.stderr
parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19a361e4df5989f78f168d53152cbe8a with result from task 164
parsl.dataflow.dflow DEBUG: Task 164 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/fc03c504-c44c-4f92-a148-20cc496aaeef_isrForCrosstalkSources_946686_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/fc03c504-c44c-4f92-a148-20cc496aaeef_isrForCrosstalkSources_946686_47.stderr
parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4782818124f9f7291d6e9c2fab8e084e with result from task 167
parsl.dataflow.dflow DEBUG: Task 167 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4fb831f6-ed95-4aad-bd49-fd2940b15e46_isrForCrosstalkSources_946687_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/4fb831f6-ed95-4aad-bd49-fd2940b15e46_isrForCrosstalkSources_946687_24.stderr
parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67f14207f12887210e9ee47da454bbd3 with result from task 166
parsl.dataflow.dflow DEBUG: Task 166 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/168a4ecd-e58d-4f74-ab0a-4872cd76f81b_isrForCrosstalkSources_946688_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946688/168a4ecd-e58d-4f74-ab0a-4872cd76f81b_isrForCrosstalkSources_946688_52.stderr
parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2aa8f873681dc28364227f81d0c2258 with result from task 168
parsl.dataflow.dflow DEBUG: Task 168 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/56d6ea46-1589-4853-9a03-78238503cd14_isrForCrosstalkSources_946687_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/56d6ea46-1589-4853-9a03-78238503cd14_isrForCrosstalkSources_946687_38.stderr
parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b7e53177bcc4adb8575613c9bfa71ba0 with result from task 181
parsl.dataflow.dflow DEBUG: Task 181 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fcf2f95e-37f9-4364-b910-67335135bc87_isrForCrosstalkSources_946687_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fcf2f95e-37f9-4364-b910-67335135bc87_isrForCrosstalkSources_946687_10.stderr
parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 206dd764be13139002b2dc9c9cd02f47 with result from task 184
parsl.dataflow.dflow DEBUG: Task 184 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/c4265395-b27a-458b-b2e9-7e9d3688f16d_isrForCrosstalkSources_946686_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/c4265395-b27a-458b-b2e9-7e9d3688f16d_isrForCrosstalkSources_946686_44.stderr
2024-03-14 02:52:56 proc_lsst.multi:146 [INFO]  found job 13859 in provider local
proc_lsst.multi INFO: found job 13859 in provider local
2024-03-14 02:52:56 proc_lsst.multi:146 [INFO]  found job 17028877 in provider astro
proc_lsst.multi INFO: found job 17028877 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:56 proc_lsst.multi:146 [INFO]  found job 17028878 in provider astro
proc_lsst.multi INFO: found job 17028878 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028879 in provider astro
proc_lsst.multi INFO: found job 17028879 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028880 in provider astro
proc_lsst.multi INFO: found job 17028880 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028881 in provider astro
proc_lsst.multi INFO: found job 17028881 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e20378d25cfe362cb1db4100b1172598 with result from task 165
parsl.dataflow.dflow DEBUG: Task 165 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/27feb194-1822-4d31-9d11-867cfe20a62f_isrForCrosstalkSources_946687_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/27feb194-1822-4d31-9d11-867cfe20a62f_isrForCrosstalkSources_946687_44.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028882 in provider astro
proc_lsst.multi INFO: found job 17028882 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028883 in provider astro
proc_lsst.multi INFO: found job 17028883 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028884 in provider astro
proc_lsst.multi INFO: found job 17028884 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028885 in provider astro
proc_lsst.multi INFO: found job 17028885 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028886 in provider astro
proc_lsst.multi INFO: found job 17028886 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028887 in provider astro
proc_lsst.multi INFO: found job 17028887 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:57 proc_lsst.multi:146 [INFO]  found job 17028888 in provider astro
proc_lsst.multi INFO: found job 17028888 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:58 proc_lsst.multi:146 [INFO]  found job 17028889 in provider astro
proc_lsst.multi INFO: found job 17028889 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:58 proc_lsst.multi:146 [INFO]  found job 17028890 in provider astro
proc_lsst.multi INFO: found job 17028890 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:58 proc_lsst.multi:146 [INFO]  found job 17028891 in provider astro
proc_lsst.multi INFO: found job 17028891 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:58 proc_lsst.multi:146 [INFO]  found job 17028892 in provider astro
proc_lsst.multi INFO: found job 17028892 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:58 proc_lsst.multi:146 [INFO]  found job 17028893 in provider astro
proc_lsst.multi INFO: found job 17028893 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:58 proc_lsst.multi:146 [INFO]  found job 17028894 in provider astro
proc_lsst.multi INFO: found job 17028894 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:58 proc_lsst.multi:146 [INFO]  found job 17028895 in provider astro
proc_lsst.multi INFO: found job 17028895 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:58 proc_lsst.multi:146 [INFO]  found job 17028896 in provider astro
proc_lsst.multi INFO: found job 17028896 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:58 proc_lsst.multi:146 [INFO]  found job 17028897 in provider astro
proc_lsst.multi INFO: found job 17028897 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:59 proc_lsst.multi:146 [INFO]  found job 17028898 in provider astro
proc_lsst.multi INFO: found job 17028898 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:59 proc_lsst.multi:146 [INFO]  found job 17028899 in provider astro
proc_lsst.multi INFO: found job 17028899 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:59 proc_lsst.multi:146 [INFO]  found job 17028900 in provider astro
proc_lsst.multi INFO: found job 17028900 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:59 proc_lsst.multi:146 [INFO]  found job 17028901 in provider astro
proc_lsst.multi INFO: found job 17028901 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:59 proc_lsst.multi:146 [INFO]  found job 17028902 in provider astro
proc_lsst.multi INFO: found job 17028902 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:59 proc_lsst.multi:146 [INFO]  found job 17028903 in provider astro
proc_lsst.multi INFO: found job 17028903 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:59 proc_lsst.multi:146 [INFO]  found job 17028904 in provider astro
proc_lsst.multi INFO: found job 17028904 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:59 proc_lsst.multi:146 [INFO]  found job 17028905 in provider astro
proc_lsst.multi INFO: found job 17028905 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c0947eaf9206710a7290b3dd53634e6 with result from task 170
parsl.dataflow.dflow DEBUG: Task 170 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fcc109fb-d4ab-4915-a7a9-7f65ab131aea_isrForCrosstalkSources_946687_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946687/fcc109fb-d4ab-4915-a7a9-7f65ab131aea_isrForCrosstalkSources_946687_39.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:52:59 proc_lsst.multi:146 [INFO]  found job 17028906 in provider astro
proc_lsst.multi INFO: found job 17028906 in provider astro
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028877,17028878,17028879,17028880,17028881,17028882,17028883,17028884,17028885,17028886,17028887,17028888,17028889,17028890,17028891,17028892,17028893,17028894,17028895,17028896,17028897,17028898,17028899,17028900,17028901,17028902,17028903,17028904,17028905,17028906'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028904 R
17028905 R
17028906 R
17028896 R
17028897 R
17028898 R
17028899 R
17028900 R
17028901 R
17028902 R
17028903 R
17028890 R
17028891 R
17028892 R
17028893 R
17028894 R
17028895 R
17028886 R
17028887 R
17028888 R
17028889 R
17028885 R
17028879 R
17028880 R
17028881 R
17028882 R
17028883 R
17028884 R
17028877 R
17028878 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028904 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028905 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028906 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028896 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028897 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028898 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028899 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028900 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028901 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028902 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028903 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028890 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028891 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028892 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028893 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028894 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028895 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028886 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028887 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028888 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028889 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028885 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028879 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028880 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028881 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028882 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028883 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028884 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028877 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028878 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:00 proc_lsst.multi:146 [INFO]  found job 17028908 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028908 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:00 proc_lsst.multi:146 [INFO]  found job 17028909 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028909 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:00 proc_lsst.multi:146 [INFO]  found job 17028910 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028910 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:00 proc_lsst.multi:146 [INFO]  found job 17028911 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028911 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:00 proc_lsst.multi:146 [INFO]  found job 17028912 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028912 in provider escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028908,17028909,17028910,17028911,17028912'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028912 R
17028908 R
17028909 R
17028910 R
17028911 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028912 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028908 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028909 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028910 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028911 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:00 proc_lsst.multi:146 [INFO]  found job 17028913 in provider escience_a40
proc_lsst.multi INFO: found job 17028913 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:00 proc_lsst.multi:146 [INFO]  found job 17028914 in provider escience_a40
proc_lsst.multi INFO: found job 17028914 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:00 proc_lsst.multi:146 [INFO]  found job 17028915 in provider escience_a40
proc_lsst.multi INFO: found job 17028915 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:00 proc_lsst.multi:146 [INFO]  found job 17028916 in provider escience_a40
proc_lsst.multi INFO: found job 17028916 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1644b896c8dbfe00e1a27b20bd88a994 with result from task 174
parsl.dataflow.dflow DEBUG: Task 174 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cd68e496-0d53-46a1-a35b-3f3a00592ca1_isrForCrosstalkSources_946686_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/cd68e496-0d53-46a1-a35b-3f3a00592ca1_isrForCrosstalkSources_946686_55.stderr
parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 41c8a168f641767111ef08e1533e4858 with result from task 172
parsl.dataflow.dflow DEBUG: Task 172 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/87e94383-1fd3-4a08-b337-b8699219a576_isrForCrosstalkSources_946686_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/87e94383-1fd3-4a08-b337-b8699219a576_isrForCrosstalkSources_946686_3.stderr
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028917 in provider escience_a40
proc_lsst.multi INFO: found job 17028917 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028918 in provider escience_a40
proc_lsst.multi INFO: found job 17028918 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9fcb9ba7f9b8692fe9553715f449023d with result from task 171
parsl.dataflow.dflow INFO: DFK cleanup initiated
parsl.dataflow.dflow INFO: Summary of tasks in DFK:
parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0
parsl.dataflow.dflow INFO: Tasks in state States.pending: 0
parsl.dataflow.dflow INFO: Tasks in state States.running: 0
parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 186
parsl.dataflow.dflow INFO: Tasks in state States.failed: 0
parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0
parsl.dataflow.dflow INFO: Tasks in state States.launched: 0
parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0
parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0
parsl.dataflow.dflow INFO: Tasks in state States.joining: 0
parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0
parsl.dataflow.dflow INFO: End of summary
parsl.dataflow.dflow DEBUG: Task 171 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9e20474d-0e3c-416f-b881-c275652fac32_isrForCrosstalkSources_946686_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/submit/DEEP/20201018/B1b/science#step0/20240314T025028Z/logs/isrForCrosstalkSources/946686/9e20474d-0e3c-416f-b881-c275652fac32_isrForCrosstalkSources_946686_43.stderr
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Closing job status poller
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028919 in provider escience_a40
proc_lsst.multi INFO: found job 17028919 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028920 in provider escience_a40
proc_lsst.multi INFO: found job 17028920 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028921 in provider escience_a40
proc_lsst.multi INFO: found job 17028921 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028922 in provider escience_a40
proc_lsst.multi INFO: found job 17028922 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028923 in provider escience_a40
proc_lsst.multi INFO: found job 17028923 in provider escience_a40
parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17028913,17028914,17028915,17028916,17028917,17028918,17028919,17028920,17028921,17028922,17028923'
parsl.providers.slurm.slurm DEBUG: squeue returned 17028922 R
17028923 R
17028913 R
17028914 R
17028915 R
17028916 R
17028917 R
17028918 R
17028919 R
17028920 R
17028921 R
 
parsl.providers.slurm.slurm DEBUG: Updating job 17028922 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028923 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028913 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028914 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028915 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028916 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028917 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028918 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028919 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028920 with slurm status R to parsl state JobState.RUNNING
parsl.providers.slurm.slurm DEBUG: Updating job 17028921 with slurm status R to parsl state JobState.RUNNING
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 0
parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 47/0 running/pending blocks, and 48 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks
parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (47) than minimum blocks (0)
parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in
parsl.dataflow.strategy DEBUG: Idle time 3.0994415283203125e-06s is less than max_idletime 120.0s for executor multi; not scaling in
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23253610635600
parsl.dataflow.dflow INFO: Terminated job status poller
parsl.dataflow.dflow INFO: Scaling in and shutting down executors
parsl.dataflow.dflow INFO: Scaling in executor multi
parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=47, block_ids=[]
parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 47 blocks
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 39bdc25f17a0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 39bdc25f17a0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d0f277077268
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d0f277077268
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5c554e12b4ec
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5c554e12b4ec
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b7ecee85483f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b7ecee85483f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd085b8bed5b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd085b8bed5b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d3afce092df7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d3afce092df7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 195aa4f2e929
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 195aa4f2e929
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7d1661a4c0c2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7d1661a4c0c2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 14de72468dd9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 14de72468dd9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: de08ca1b29b3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: de08ca1b29b3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4e92de47d27c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4e92de47d27c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b745053505a9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b745053505a9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d4e926e7b0e1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d4e926e7b0e1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4123971b6ebe
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4123971b6ebe
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe87c99ea492
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe87c99ea492
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3d7644cdf87b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3d7644cdf87b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4cfecde90c46
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4cfecde90c46
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5d52b75e2123
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5d52b75e2123
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d7912ccf09f5
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d7912ccf09f5
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1a1e2f870941
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1a1e2f870941
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5532510f2b25
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5532510f2b25
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e94b6732fc9a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e94b6732fc9a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2b08d88d331a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2b08d88d331a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d10060660b1
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d10060660b1
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 76655c410276
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 76655c410276
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3c1d920e75a8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3c1d920e75a8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 269e8a6c5ee4
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 269e8a6c5ee4
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d6378f5be3ad
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d6378f5be3ad
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fd4dd195ce63
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fd4dd195ce63
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7ea439a9228d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7ea439a9228d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cbea58432542
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cbea58432542
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d7cc66dd2ba3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d7cc66dd2ba3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2fc51241ffe2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2fc51241ffe2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ea1da22d4e0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ea1da22d4e0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7da18b565062
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7da18b565062
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eb736f2ec299
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eb736f2ec299
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f0a15e4f2a8a
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f0a15e4f2a8a
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 861f6e5f8fa0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 861f6e5f8fa0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b93108f1b603
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b93108f1b603
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60318f5a3d5c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60318f5a3d5c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2b2add2b4b42
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2b2add2b4b42
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bd6202cf37a8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bd6202cf37a8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9c3807d9ec20
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9c3807d9ec20
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 88ac9b760a2b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 88ac9b760a2b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 350daf1596f8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 350daf1596f8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 85d407a2e7db
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 85d407a2e7db
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2d7f9e5fd68b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2d7f9e5fd68b
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028909 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028909 in provider escience_rtx6k
2024-03-14 02:53:01 proc_lsst.multi:201 [INFO]  cancelling 17028909 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028909 on provider escience_rtx6k
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028908 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028908 in provider escience_rtx6k
2024-03-14 02:53:01 proc_lsst.multi:201 [INFO]  cancelling 17028908 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028908 on provider escience_rtx6k
2024-03-14 02:53:01 proc_lsst.multi:146 [INFO]  found job 17028911 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028911 in provider escience_rtx6k
2024-03-14 02:53:01 proc_lsst.multi:201 [INFO]  cancelling 17028911 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028911 on provider escience_rtx6k
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028910 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028910 in provider escience_rtx6k
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028910 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028910 on provider escience_rtx6k
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028912 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028912 in provider escience_rtx6k
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028912 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028912 on provider escience_rtx6k
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028884 in provider astro
proc_lsst.multi INFO: found job 17028884 in provider astro
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028884 on provider astro
proc_lsst.multi INFO: cancelling 17028884 on provider astro
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028888 in provider astro
proc_lsst.multi INFO: found job 17028888 in provider astro
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028888 on provider astro
proc_lsst.multi INFO: cancelling 17028888 on provider astro
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028900 in provider astro
proc_lsst.multi INFO: found job 17028900 in provider astro
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028900 on provider astro
proc_lsst.multi INFO: cancelling 17028900 on provider astro
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028904 in provider astro
proc_lsst.multi INFO: found job 17028904 in provider astro
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028904 on provider astro
proc_lsst.multi INFO: cancelling 17028904 on provider astro
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028901 in provider astro
proc_lsst.multi INFO: found job 17028901 in provider astro
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028901 on provider astro
proc_lsst.multi INFO: cancelling 17028901 on provider astro
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028916 in provider escience_a40
proc_lsst.multi INFO: found job 17028916 in provider escience_a40
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028916 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028916 on provider escience_a40
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028917 in provider escience_a40
proc_lsst.multi INFO: found job 17028917 in provider escience_a40
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028917 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028917 on provider escience_a40
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028920 in provider escience_a40
proc_lsst.multi INFO: found job 17028920 in provider escience_a40
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028920 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028920 on provider escience_a40
2024-03-14 02:53:02 proc_lsst.multi:146 [INFO]  found job 17028918 in provider escience_a40
proc_lsst.multi INFO: found job 17028918 in provider escience_a40
2024-03-14 02:53:02 proc_lsst.multi:201 [INFO]  cancelling 17028918 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028918 on provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028902 in provider astro
proc_lsst.multi INFO: found job 17028902 in provider astro
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028902 on provider astro
proc_lsst.multi INFO: cancelling 17028902 on provider astro
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028906 in provider astro
proc_lsst.multi INFO: found job 17028906 in provider astro
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028906 on provider astro
proc_lsst.multi INFO: cancelling 17028906 on provider astro
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028915 in provider escience_a40
proc_lsst.multi INFO: found job 17028915 in provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028915 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028915 on provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028923 in provider escience_a40
proc_lsst.multi INFO: found job 17028923 in provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028923 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028923 on provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028914 in provider escience_a40
proc_lsst.multi INFO: found job 17028914 in provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028914 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028914 on provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028881 in provider astro
proc_lsst.multi INFO: found job 17028881 in provider astro
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028881 on provider astro
proc_lsst.multi INFO: cancelling 17028881 on provider astro
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028921 in provider escience_a40
proc_lsst.multi INFO: found job 17028921 in provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028921 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028921 on provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028913 in provider escience_a40
proc_lsst.multi INFO: found job 17028913 in provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028913 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028913 on provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028922 in provider escience_a40
proc_lsst.multi INFO: found job 17028922 in provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028922 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028922 on provider escience_a40
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028880 in provider astro
proc_lsst.multi INFO: found job 17028880 in provider astro
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028880 on provider astro
proc_lsst.multi INFO: cancelling 17028880 on provider astro
2024-03-14 02:53:03 proc_lsst.multi:146 [INFO]  found job 17028887 in provider astro
proc_lsst.multi INFO: found job 17028887 in provider astro
2024-03-14 02:53:03 proc_lsst.multi:201 [INFO]  cancelling 17028887 on provider astro
proc_lsst.multi INFO: cancelling 17028887 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028883 in provider astro
proc_lsst.multi INFO: found job 17028883 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028883 on provider astro
proc_lsst.multi INFO: cancelling 17028883 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028885 in provider astro
proc_lsst.multi INFO: found job 17028885 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028885 on provider astro
proc_lsst.multi INFO: cancelling 17028885 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028882 in provider astro
proc_lsst.multi INFO: found job 17028882 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028882 on provider astro
proc_lsst.multi INFO: cancelling 17028882 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028879 in provider astro
proc_lsst.multi INFO: found job 17028879 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028879 on provider astro
proc_lsst.multi INFO: cancelling 17028879 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028919 in provider escience_a40
proc_lsst.multi INFO: found job 17028919 in provider escience_a40
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028919 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028919 on provider escience_a40
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 13859 in provider local
proc_lsst.multi INFO: found job 13859 in provider local
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 13859 on provider local
proc_lsst.multi INFO: cancelling 13859 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 13859
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028892 in provider astro
proc_lsst.multi INFO: found job 17028892 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028892 on provider astro
proc_lsst.multi INFO: cancelling 17028892 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028898 in provider astro
proc_lsst.multi INFO: found job 17028898 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028898 on provider astro
proc_lsst.multi INFO: cancelling 17028898 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028896 in provider astro
proc_lsst.multi INFO: found job 17028896 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028896 on provider astro
proc_lsst.multi INFO: cancelling 17028896 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028889 in provider astro
proc_lsst.multi INFO: found job 17028889 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028889 on provider astro
proc_lsst.multi INFO: cancelling 17028889 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028899 in provider astro
proc_lsst.multi INFO: found job 17028899 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028899 on provider astro
proc_lsst.multi INFO: cancelling 17028899 on provider astro
2024-03-14 02:53:04 proc_lsst.multi:146 [INFO]  found job 17028894 in provider astro
proc_lsst.multi INFO: found job 17028894 in provider astro
2024-03-14 02:53:04 proc_lsst.multi:201 [INFO]  cancelling 17028894 on provider astro
proc_lsst.multi INFO: cancelling 17028894 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028893 in provider astro
proc_lsst.multi INFO: found job 17028893 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028893 on provider astro
proc_lsst.multi INFO: cancelling 17028893 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028890 in provider astro
proc_lsst.multi INFO: found job 17028890 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028890 on provider astro
proc_lsst.multi INFO: cancelling 17028890 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028895 in provider astro
proc_lsst.multi INFO: found job 17028895 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028895 on provider astro
proc_lsst.multi INFO: cancelling 17028895 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028891 in provider astro
proc_lsst.multi INFO: found job 17028891 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028891 on provider astro
proc_lsst.multi INFO: cancelling 17028891 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028897 in provider astro
proc_lsst.multi INFO: found job 17028897 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028897 on provider astro
proc_lsst.multi INFO: cancelling 17028897 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028903 in provider astro
proc_lsst.multi INFO: found job 17028903 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028903 on provider astro
proc_lsst.multi INFO: cancelling 17028903 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028905 in provider astro
proc_lsst.multi INFO: found job 17028905 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028905 on provider astro
proc_lsst.multi INFO: cancelling 17028905 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028878 in provider astro
proc_lsst.multi INFO: found job 17028878 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028878 on provider astro
proc_lsst.multi INFO: cancelling 17028878 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028886 in provider astro
proc_lsst.multi INFO: found job 17028886 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028886 on provider astro
proc_lsst.multi INFO: cancelling 17028886 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028877 in provider astro
proc_lsst.multi INFO: found job 17028877 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028877 on provider astro
proc_lsst.multi INFO: cancelling 17028877 on provider astro
parsl.dataflow.dflow INFO: Shutting down executor multi
2024-03-14 02:53:05 proc_lsst.multi:40 [INFO]  Cancelling all provider resources
proc_lsst.multi INFO: Cancelling all provider resources
2024-03-14 02:53:05 proc_lsst.multi:47 [INFO]  new jobs since last cancel ['17028877', '17028878', '17028879', '17028880', '17028881', '17028882', '17028883', '17028884', '17028885', '17028886', '17028887', '17028888', '17028889', '17028890', '17028891', '17028892', '17028893', '17028894', '17028895', '17028896', '17028897', '17028898', '17028899', '17028900', '17028901', '17028902', '17028903', '17028904', '17028905', '17028906', '17028913', '17028914', '17028915', '17028916', '17028917', '17028918', '17028919', '17028920', '17028921', '17028922', '17028923', '17028908', '17028909', '17028910', '17028911', '17028912', '13859']
proc_lsst.multi INFO: new jobs since last cancel ['17028877', '17028878', '17028879', '17028880', '17028881', '17028882', '17028883', '17028884', '17028885', '17028886', '17028887', '17028888', '17028889', '17028890', '17028891', '17028892', '17028893', '17028894', '17028895', '17028896', '17028897', '17028898', '17028899', '17028900', '17028901', '17028902', '17028903', '17028904', '17028905', '17028906', '17028913', '17028914', '17028915', '17028916', '17028917', '17028918', '17028919', '17028920', '17028921', '17028922', '17028923', '17028908', '17028909', '17028910', '17028911', '17028912', '13859']
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028877 in provider astro
proc_lsst.multi INFO: found job 17028877 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028877 on provider astro
proc_lsst.multi INFO: cancelling 17028877 on provider astro
2024-03-14 02:53:05 proc_lsst.multi:146 [INFO]  found job 17028878 in provider astro
proc_lsst.multi INFO: found job 17028878 in provider astro
2024-03-14 02:53:05 proc_lsst.multi:201 [INFO]  cancelling 17028878 on provider astro
proc_lsst.multi INFO: cancelling 17028878 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028879 in provider astro
proc_lsst.multi INFO: found job 17028879 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028879 on provider astro
proc_lsst.multi INFO: cancelling 17028879 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028880 in provider astro
proc_lsst.multi INFO: found job 17028880 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028880 on provider astro
proc_lsst.multi INFO: cancelling 17028880 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028881 in provider astro
proc_lsst.multi INFO: found job 17028881 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028881 on provider astro
proc_lsst.multi INFO: cancelling 17028881 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028882 in provider astro
proc_lsst.multi INFO: found job 17028882 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028882 on provider astro
proc_lsst.multi INFO: cancelling 17028882 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028883 in provider astro
proc_lsst.multi INFO: found job 17028883 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028883 on provider astro
proc_lsst.multi INFO: cancelling 17028883 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028884 in provider astro
proc_lsst.multi INFO: found job 17028884 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028884 on provider astro
proc_lsst.multi INFO: cancelling 17028884 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028885 in provider astro
proc_lsst.multi INFO: found job 17028885 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028885 on provider astro
proc_lsst.multi INFO: cancelling 17028885 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028886 in provider astro
proc_lsst.multi INFO: found job 17028886 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028886 on provider astro
proc_lsst.multi INFO: cancelling 17028886 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028887 in provider astro
proc_lsst.multi INFO: found job 17028887 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028887 on provider astro
proc_lsst.multi INFO: cancelling 17028887 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028888 in provider astro
proc_lsst.multi INFO: found job 17028888 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028888 on provider astro
proc_lsst.multi INFO: cancelling 17028888 on provider astro
2024-03-14 02:53:06 proc_lsst.multi:146 [INFO]  found job 17028889 in provider astro
proc_lsst.multi INFO: found job 17028889 in provider astro
2024-03-14 02:53:06 proc_lsst.multi:201 [INFO]  cancelling 17028889 on provider astro
proc_lsst.multi INFO: cancelling 17028889 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028890 in provider astro
proc_lsst.multi INFO: found job 17028890 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028890 on provider astro
proc_lsst.multi INFO: cancelling 17028890 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028891 in provider astro
proc_lsst.multi INFO: found job 17028891 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028891 on provider astro
proc_lsst.multi INFO: cancelling 17028891 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028892 in provider astro
proc_lsst.multi INFO: found job 17028892 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028892 on provider astro
proc_lsst.multi INFO: cancelling 17028892 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028893 in provider astro
proc_lsst.multi INFO: found job 17028893 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028893 on provider astro
proc_lsst.multi INFO: cancelling 17028893 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028894 in provider astro
proc_lsst.multi INFO: found job 17028894 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028894 on provider astro
proc_lsst.multi INFO: cancelling 17028894 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028895 in provider astro
proc_lsst.multi INFO: found job 17028895 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028895 on provider astro
proc_lsst.multi INFO: cancelling 17028895 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028896 in provider astro
proc_lsst.multi INFO: found job 17028896 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028896 on provider astro
proc_lsst.multi INFO: cancelling 17028896 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028897 in provider astro
proc_lsst.multi INFO: found job 17028897 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028897 on provider astro
proc_lsst.multi INFO: cancelling 17028897 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028898 in provider astro
proc_lsst.multi INFO: found job 17028898 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028898 on provider astro
proc_lsst.multi INFO: cancelling 17028898 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028899 in provider astro
proc_lsst.multi INFO: found job 17028899 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028899 on provider astro
proc_lsst.multi INFO: cancelling 17028899 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028900 in provider astro
proc_lsst.multi INFO: found job 17028900 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028900 on provider astro
proc_lsst.multi INFO: cancelling 17028900 on provider astro
2024-03-14 02:53:07 proc_lsst.multi:146 [INFO]  found job 17028901 in provider astro
proc_lsst.multi INFO: found job 17028901 in provider astro
2024-03-14 02:53:07 proc_lsst.multi:201 [INFO]  cancelling 17028901 on provider astro
proc_lsst.multi INFO: cancelling 17028901 on provider astro
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028902 in provider astro
proc_lsst.multi INFO: found job 17028902 in provider astro
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028902 on provider astro
proc_lsst.multi INFO: cancelling 17028902 on provider astro
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028903 in provider astro
proc_lsst.multi INFO: found job 17028903 in provider astro
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028903 on provider astro
proc_lsst.multi INFO: cancelling 17028903 on provider astro
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028904 in provider astro
proc_lsst.multi INFO: found job 17028904 in provider astro
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028904 on provider astro
proc_lsst.multi INFO: cancelling 17028904 on provider astro
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028905 in provider astro
proc_lsst.multi INFO: found job 17028905 in provider astro
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028905 on provider astro
proc_lsst.multi INFO: cancelling 17028905 on provider astro
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028906 in provider astro
proc_lsst.multi INFO: found job 17028906 in provider astro
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028906 on provider astro
proc_lsst.multi INFO: cancelling 17028906 on provider astro
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028913 in provider escience_a40
proc_lsst.multi INFO: found job 17028913 in provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028913 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028913 on provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028914 in provider escience_a40
proc_lsst.multi INFO: found job 17028914 in provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028914 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028914 on provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028915 in provider escience_a40
proc_lsst.multi INFO: found job 17028915 in provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028915 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028915 on provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028916 in provider escience_a40
proc_lsst.multi INFO: found job 17028916 in provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028916 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028916 on provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028917 in provider escience_a40
proc_lsst.multi INFO: found job 17028917 in provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028917 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028917 on provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:146 [INFO]  found job 17028918 in provider escience_a40
proc_lsst.multi INFO: found job 17028918 in provider escience_a40
2024-03-14 02:53:08 proc_lsst.multi:201 [INFO]  cancelling 17028918 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028918 on provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028919 in provider escience_a40
proc_lsst.multi INFO: found job 17028919 in provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028919 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028919 on provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028920 in provider escience_a40
proc_lsst.multi INFO: found job 17028920 in provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028920 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028920 on provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028921 in provider escience_a40
proc_lsst.multi INFO: found job 17028921 in provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028921 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028921 on provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028922 in provider escience_a40
proc_lsst.multi INFO: found job 17028922 in provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028922 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028922 on provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028923 in provider escience_a40
proc_lsst.multi INFO: found job 17028923 in provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028923 on provider escience_a40
proc_lsst.multi INFO: cancelling 17028923 on provider escience_a40
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028908 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028908 in provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028908 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028908 on provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028909 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028909 in provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028909 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028909 on provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028910 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028910 in provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028910 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028910 on provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028911 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028911 in provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028911 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028911 on provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 17028912 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17028912 in provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 17028912 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17028912 on provider escience_rtx6k
2024-03-14 02:53:09 proc_lsst.multi:146 [INFO]  found job 13859 in provider local
proc_lsst.multi INFO: found job 13859 in provider local
2024-03-14 02:53:09 proc_lsst.multi:201 [INFO]  cancelling 13859 on provider local
proc_lsst.multi INFO: cancelling 13859 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 13859
parsl.providers.local.local WARNING: Failed to kill PID: 13859 and child processes on local
2024-03-14 02:53:10 proc_lsst.multi:50 [INFO]  no new jobs since last cancel, resuming executor shutdown
proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown
parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown
parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt
parsl.dataflow.dflow INFO: Shut down executor multi
parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal
parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete
parsl.executors.threads DEBUG: Done with executor shutdown
parsl.dataflow.dflow INFO: Shut down executor _parsl_internal
parsl.dataflow.dflow INFO: Terminated executors
parsl.dataflow.dflow INFO: DFK cleanup complete
parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread
lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 134.5263 seconds
lsst.ctrl.bps.drivers INFO: Run 'DEEP_20201018_B1b_science#step0_20240314T025028Z' submitted for execution with id 'None'
lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 134.5323 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.002 Gibyte
lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 157.8769 seconds; current memory usage: 0.272 Gibyte, delta: 0.135 Gibyte, peak delta: 0.137 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.274 Gibyte (largest child process)
Run Id: None
Run Name: DEEP_20201018_B1b_science#step0_20240314T025028Z
parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up

real	2m39.678s
user	0m31.675s
sys	0m14.225s