+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit
+ bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210510/A1h --output-run DEEP/20210510/A1h/science#step0/20240315T102059Z --qgraph pipeline.qgraph
lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process.
lsst.ctrl.bps.drivers INFO: Starting submission process
lsst.ctrl.bps.drivers INFO: Initializing execution environment
lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 6.0564 seconds; current memory usage: 0.181 Gibyte, delta: 0.044 Gibyte, peak delta: 0.054 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph)
lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.2703 seconds
lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.2822 seconds
lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/pipeline_orig.qgraph DEEP/20210510/A1h/science#step0/20240315T102059Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/pipeline.qgraph 
lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/pipeline.qgraph'
lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.5811 seconds
lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 7.6231 seconds; current memory usage: 0.273 Gibyte, delta: 0.091 Gibyte, peak delta: 0.082 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.266 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs)
lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0127 seconds; current memory usage: 0.273 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.266 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 372 cluster(s)
lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow)
lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210510_A1h_science#step0_20240315T102059Z'
lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.3626 seconds; current memory usage: 0.273 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.266 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 374 job(s) (including final)
lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow)
parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2
Traceback (most recent call last):
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses
    s_addresses.add(address_by_interface(interface))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper
    retval = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface
    return socket.inet_ntoa(fcntl.ioctl(
                            ^^^^^^^^^^^^
OSError: [Errno 99] Cannot assign requested address
lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z
lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 1.1468 seconds; current memory usage: 0.274 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.266 Gibyte (largest child process)
lsst.ctrl.bps.drivers INFO: Starting submit stage
lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution
parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000
parsl.dataflow.dflow INFO: Starting DataFlowKernel with config
Config(
    app_cache=True, 
    checkpoint_files=None, 
    checkpoint_mode='task_exit', 
    checkpoint_period=None, 
    executors=[MultiHighThroughputExecutor()], 
    garbage_collect=True, 
    initialize_logging=True, 
    internal_tasks_max_threads=10, 
    max_idletime=120.0, 
    monitoring=None, 
    retries=1, 
    retry_handler=None, 
    run_dir='runinfo', 
    strategy='simple', 
    usage_tracking=False
)
parsl.dataflow.dflow INFO: Parsl version: 2023.06.12
parsl.usage_tracking.usage DEBUG: Tracking status: False
parsl.dataflow.dflow INFO: Run id is: e9df928e-a535-4cf6-a2d9-0bc8290a0ffe
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py
parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py
parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name
parsl.dataflow.memoization INFO: App caching initialized
parsl.dataflow.strategy DEBUG: Scaling strategy: simple
parsl.executors.high_throughput.executor DEBUG: Starting queue management thread
parsl.executors.high_throughput.executor DEBUG: queue management worker starting
parsl.executors.high_throughput.executor DEBUG: Started queue management thread
Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z
parsl.executors.high_throughput.executor DEBUG: Created management thread: <Thread(HTEX-Queue-Management-Thread, started daemon 22647805454080)>
parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id={block_id} --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider:
<proc_lsst.multi.MultiProvider object at 0x14991b8dd950>
parsl.executors.status_handling INFO: Scaling out by 1 blocks
parsl.executors.status_handling INFO: Allocated block ID 0
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0
2024-03-15 10:21:32 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-0
2024-03-15 10:21:32 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:32 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-15 10:21:32 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:32 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=0 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2 to local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22647872635152
parsl.providers.local.local DEBUG: Launching in remote mode
2024-03-15 10:21:33 proc_lsst.multi:170 [INFO]  [multi] job_id 6561
proc_lsst.multi INFO: [multi] job_id 6561
2024-03-15 10:21:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:33 proc_lsst.multi:178 [INFO]  [multi] provider local accepted submit and returned 6561
proc_lsst.multi INFO: [multi] provider local accepted submit and returned 6561
parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 6561
parsl.dataflow.job_status_poller DEBUG: Adding executor multi
parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: <AppFuture at 0x14991bb5f390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 93fca73b61f92b257568c855bb77bbcc
parsl.dataflow.memoization INFO: Task 0 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a98a02a2-2d9f-4666-b195-3c2d2a12bca2_isrForCrosstalkSources_995994_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a98a02a2-2d9f-4666-b195-3c2d2a12bca2_isrForCrosstalkSources_995994_43.stderr
parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: <AppFuture at 0x14991b8fcc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 1 has memoization hash c01ad9bf7dca305c258023e8779f134e
parsl.dataflow.memoization INFO: Task 1 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8a98e235-528f-42ed-83cc-18ae44abd87b_isrForCrosstalkSources_995850_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8a98e235-528f-42ed-83cc-18ae44abd87b_isrForCrosstalkSources_995850_34.stderr
parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: <AppFuture at 0x14991bacbc50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 2 has memoization hash b6a405c159bb1a99dbae7833c871f2c0
parsl.dataflow.memoization INFO: Task 2 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e3e9ff59-3685-470c-8e76-d85b723e8f6c_isrForCrosstalkSources_995994_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e3e9ff59-3685-470c-8e76-d85b723e8f6c_isrForCrosstalkSources_995994_6.stderr
parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: <AppFuture at 0x14991ba9ced0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 96678af876b922f54a53c4d0209a3d82
parsl.dataflow.memoization INFO: Task 3 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9358a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5483a64b-3542-4e8a-99a9-904074e433a3_isrForCrosstalkSources_995994_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5483a64b-3542-4e8a-99a9-904074e433a3_isrForCrosstalkSources_995994_14.stderr
parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: <AppFuture at 0x14991ba9e3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 4 has memoization hash ee8518200a50ba986d79b3d01c72a1e5
parsl.dataflow.memoization INFO: Task 4 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b935b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/de3b13ac-6226-4951-a2cd-2209dbbb155a_isrForCrosstalkSources_995849_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/de3b13ac-6226-4951-a2cd-2209dbbb155a_isrForCrosstalkSources_995849_57.stderr
parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: <AppFuture at 0x14991bc42b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 32ee35563fa01e24ea0f7837da281ba1
parsl.dataflow.memoization INFO: Task 5 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b935940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dd6aed3c-4c1a-4334-8e3a-6324a571de4c_isrForCrosstalkSources_995848_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dd6aed3c-4c1a-4334-8e3a-6324a571de4c_isrForCrosstalkSources_995848_51.stderr
parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: <AppFuture at 0x14991bc40e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 6 has memoization hash f09d78ebbbc0d628a1121f3b32f5f27d
parsl.dataflow.memoization INFO: Task 6 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b935bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/4f0b73b7-567d-44dc-8e03-ea224a8ca940_isrForCrosstalkSources_995848_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/4f0b73b7-567d-44dc-8e03-ea224a8ca940_isrForCrosstalkSources_995848_18.stderr
parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: <AppFuture at 0x14991bc71210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 3aa2a4065bfed261883942e66b3185d1
parsl.dataflow.memoization INFO: Task 7 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b935260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/88bc5c54-da5f-49d9-a529-491ae4f56153_isrForCrosstalkSources_995994_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/88bc5c54-da5f-49d9-a529-491ae4f56153_isrForCrosstalkSources_995994_45.stderr
parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: <AppFuture at 0x14991bc72690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 5bc74a0b4e346020eb74838936e2c497
parsl.dataflow.memoization INFO: Task 8 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b935da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/df8a41b0-1e4c-4c2b-94e0-6e0566fa1eb1_isrForCrosstalkSources_995995_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/df8a41b0-1e4c-4c2b-94e0-6e0566fa1eb1_isrForCrosstalkSources_995995_9.stderr
parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: <AppFuture at 0x14991bc67410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 9 has memoization hash a60dad045c5cfa6ebe92799265309ea1
parsl.dataflow.memoization INFO: Task 9 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b935580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/0c959044-1328-4b2d-8534-525bdc93ceda_isrForCrosstalkSources_995850_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/0c959044-1328-4b2d-8534-525bdc93ceda_isrForCrosstalkSources_995850_16.stderr
parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: <AppFuture at 0x14991bc64710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 5649f3a1d123841140886003543e465f
parsl.dataflow.memoization INFO: Task 10 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b935a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/d639a1b1-86ab-4ffc-b0ca-6441c5d9b833_isrForCrosstalkSources_995995_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/d639a1b1-86ab-4ffc-b0ca-6441c5d9b833_isrForCrosstalkSources_995995_20.stderr
parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: <AppFuture at 0x14991bc7e610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 54cb59b074609738cb5ab99356e45c6f
parsl.dataflow.memoization INFO: Task 11 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b935d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/49facc7d-4535-4b51-9ebd-9dfcbe3935b4_isrForCrosstalkSources_995993_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/49facc7d-4535-4b51-9ebd-9dfcbe3935b4_isrForCrosstalkSources_995993_18.stderr
parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: <AppFuture at 0x14992617eb90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 12 has memoization hash d4daef194d99022b19d52d651e0f5716
parsl.dataflow.memoization INFO: Task 12 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b7cdf59f-60db-499d-b1a0-656102498b3d_isrForCrosstalkSources_995849_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b7cdf59f-60db-499d-b1a0-656102498b3d_isrForCrosstalkSources_995849_49.stderr
parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: <AppFuture at 0x14992617ef90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 13 has memoization hash ef854f12dd4ea33f3a7d92c5d8624d75
parsl.dataflow.memoization INFO: Task 13 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ba0c83f3-ccae-4a60-bdc5-112f744f152c_isrForCrosstalkSources_995994_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ba0c83f3-ccae-4a60-bdc5-112f744f152c_isrForCrosstalkSources_995994_23.stderr
parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: <AppFuture at 0x14991ba98510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 14 has memoization hash dd3d64cd8eba5217737d5e0f2077a945
parsl.dataflow.memoization INFO: Task 14 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9363e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/14ade185-6591-42a0-a6f3-774264fd335d_isrForCrosstalkSources_995848_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/14ade185-6591-42a0-a6f3-774264fd335d_isrForCrosstalkSources_995848_11.stderr
parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: <AppFuture at 0x14991bc5ced0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 8416a4c04a0e935f6ac257958f7af4f0
parsl.dataflow.memoization INFO: Task 15 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b935f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/813603c5-e9e2-46f7-9b4c-705dfbf6588b_isrForCrosstalkSources_995993_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/813603c5-e9e2-46f7-9b4c-705dfbf6588b_isrForCrosstalkSources_995993_33.stderr
parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: <AppFuture at 0x14991bc5c350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 7bac38910849186596a6e26261eca03b
parsl.dataflow.memoization INFO: Task 16 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/f2a00579-7e85-41a4-a204-e1b1a9cb2309_isrForCrosstalkSources_995850_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/f2a00579-7e85-41a4-a204-e1b1a9cb2309_isrForCrosstalkSources_995850_31.stderr
parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: <AppFuture at 0x14991bc4ba10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 35ebb73a51fe6b8014c37c29dee4e21b
parsl.dataflow.memoization INFO: Task 17 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/8025ee51-026c-4c0d-aa3c-c8e9ea5187dd_isrForCrosstalkSources_995994_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/8025ee51-026c-4c0d-aa3c-c8e9ea5187dd_isrForCrosstalkSources_995994_20.stderr
parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: <AppFuture at 0x14991bc4bc10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 18 has memoization hash cee14c6b0e95eb6ae867ae626787eb3e
parsl.dataflow.memoization INFO: Task 18 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/6f2fe2a6-a6a2-46e5-9685-20c46b79f128_isrForCrosstalkSources_995848_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/6f2fe2a6-a6a2-46e5-9685-20c46b79f128_isrForCrosstalkSources_995848_3.stderr
parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: <AppFuture at 0x14991babaa50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 595521db2919c95438cccd2c3117f042
parsl.dataflow.memoization INFO: Task 19 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9367a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c4c82d3f-507a-4a09-947d-2afb9127eff7_isrForCrosstalkSources_995993_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c4c82d3f-507a-4a09-947d-2afb9127eff7_isrForCrosstalkSources_995993_55.stderr
parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: <AppFuture at 0x14991bab9a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 20 has memoization hash a814169dda26907d45d03ba0f6d06c29
parsl.dataflow.memoization INFO: Task 20 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/5c9519b9-f6bd-41a7-84c6-3f326c5e6326_isrForCrosstalkSources_995993_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/5c9519b9-f6bd-41a7-84c6-3f326c5e6326_isrForCrosstalkSources_995993_42.stderr
parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: <AppFuture at 0x14991bc35c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 042d1dfcb81aeb524dfcb7f274bf8c11
parsl.dataflow.memoization INFO: Task 21 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/44fd7d48-eaec-4068-ab0e-3bddeb051d2e_isrForCrosstalkSources_995994_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/44fd7d48-eaec-4068-ab0e-3bddeb051d2e_isrForCrosstalkSources_995994_42.stderr
parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: <AppFuture at 0x14991bc36d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 3b1b91cd180df7c5fe76d901ce58c4dc
parsl.dataflow.memoization INFO: Task 22 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9368e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9df5789f-b2df-45d3-a967-f1aaed8dc9c5_isrForCrosstalkSources_995850_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9df5789f-b2df-45d3-a967-f1aaed8dc9c5_isrForCrosstalkSources_995850_61.stderr
parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: <AppFuture at 0x14991bc2a710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 4e5c2861832ba7bf2a6f842954478aea
parsl.dataflow.memoization INFO: Task 23 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a3327cf7-22ca-48b5-82e8-1638a3fd5ab8_isrForCrosstalkSources_995995_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a3327cf7-22ca-48b5-82e8-1638a3fd5ab8_isrForCrosstalkSources_995995_49.stderr
parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: <AppFuture at 0x14991bc2a590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 4005c83b99db7a66e73d582ddbd1f44f
parsl.dataflow.memoization INFO: Task 24 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/99e2ac32-d062-4d87-8c74-5b8260233435_isrForCrosstalkSources_995993_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/99e2ac32-d062-4d87-8c74-5b8260233435_isrForCrosstalkSources_995993_6.stderr
parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: <AppFuture at 0x14991bc1a710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 21ef0c1e6932ac3944b7ad9db4bd39a0
parsl.dataflow.memoization INFO: Task 25 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2936d3ef-e0ed-4711-9892-39e425915803_isrForCrosstalkSources_995849_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2936d3ef-e0ed-4711-9892-39e425915803_isrForCrosstalkSources_995849_9.stderr
parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: <AppFuture at 0x14991bc1b490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 9d4e37b9ff6ec44c8471717f042eb899
parsl.dataflow.memoization INFO: Task 26 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/43015391-5126-480b-becd-40bf36bf2038_isrForCrosstalkSources_995994_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/43015391-5126-480b-becd-40bf36bf2038_isrForCrosstalkSources_995994_49.stderr
parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: <AppFuture at 0x14991bc1ab90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 804cbe7d6fa13f3a55c42276c0d120a8
parsl.dataflow.memoization INFO: Task 27 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6d46ff39-c0cd-4442-9ac7-cfdbbc24767d_isrForCrosstalkSources_995993_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6d46ff39-c0cd-4442-9ac7-cfdbbc24767d_isrForCrosstalkSources_995993_21.stderr
parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: <AppFuture at 0x14992617c410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 9fce1f3960dc8d67ac02a49fb497ce76
parsl.dataflow.memoization INFO: Task 28 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/7612adc3-1638-425b-8304-667d5559bdb5_isrForCrosstalkSources_995848_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/7612adc3-1638-425b-8304-667d5559bdb5_isrForCrosstalkSources_995848_1.stderr
parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: <AppFuture at 0x14991bc0fd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 29 has memoization hash f0e4b987d34ef1c4775221f3dba17b44
parsl.dataflow.memoization INFO: Task 29 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bc704b00-aabe-4263-9c45-d17765988565_isrForCrosstalkSources_995850_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bc704b00-aabe-4263-9c45-d17765988565_isrForCrosstalkSources_995850_12.stderr
parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: <AppFuture at 0x14991bc0dd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 683a7e3e7a9a4920c266e17ca692aff8
parsl.dataflow.memoization INFO: Task 30 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/17659de3-1e1e-4617-aa45-be5a046aeff0_isrForCrosstalkSources_995848_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/17659de3-1e1e-4617-aa45-be5a046aeff0_isrForCrosstalkSources_995848_14.stderr
parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: <AppFuture at 0x14991bc19810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 72219777fa8c25a4a638f187ad3a638e
parsl.dataflow.memoization INFO: Task 31 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a1df886a-001d-4451-bb35-4c0b9e1b7605_isrForCrosstalkSources_995994_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a1df886a-001d-4451-bb35-4c0b9e1b7605_isrForCrosstalkSources_995994_46.stderr
parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: <AppFuture at 0x14991bc00d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 32 has memoization hash e18e3a326bad9cd65316d3ad60ff6cf0
parsl.dataflow.memoization INFO: Task 32 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b42081c7-2284-4d70-97c2-eef683183d11_isrForCrosstalkSources_995849_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b42081c7-2284-4d70-97c2-eef683183d11_isrForCrosstalkSources_995849_41.stderr
parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: <AppFuture at 0x14991bc01f10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 89977215285a5790a2a37e673e3d6c34
parsl.dataflow.memoization INFO: Task 33 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3a9953e2-0299-402f-9ab1-135ee708ff3d_isrForCrosstalkSources_995995_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3a9953e2-0299-402f-9ab1-135ee708ff3d_isrForCrosstalkSources_995995_22.stderr
parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: <AppFuture at 0x14991bbf21d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 793ac2168a7b85d18e0f579b7f793500
parsl.dataflow.memoization INFO: Task 34 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1587f614-38ad-4b8e-bc14-79650ac6144d_isrForCrosstalkSources_995849_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1587f614-38ad-4b8e-bc14-79650ac6144d_isrForCrosstalkSources_995849_22.stderr
parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: <AppFuture at 0x14991bbf1950 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 35 has memoization hash bb6e911e5561cf97e6aa0571ebe5f02d
parsl.dataflow.memoization INFO: Task 35 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b936de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f4ac559e-06ae-42c5-a500-84e4e53bca63_isrForCrosstalkSources_995994_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f4ac559e-06ae-42c5-a500-84e4e53bca63_isrForCrosstalkSources_995994_55.stderr
parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: <AppFuture at 0x14991bc281d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 36 has memoization hash c96197ade847efd7555c1c105ddb3564
parsl.dataflow.memoization INFO: Task 36 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fdb71650-8ddb-479b-a0cb-7537e68073bd_isrForCrosstalkSources_995995_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fdb71650-8ddb-479b-a0cb-7537e68073bd_isrForCrosstalkSources_995995_41.stderr
parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: <AppFuture at 0x14991bbe2d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 37 has memoization hash a5e2059e39956ec5e5756d65fa6c9156
parsl.dataflow.memoization INFO: Task 37 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a9b00a3d-63fc-466b-b5c8-6668eea5c551_isrForCrosstalkSources_995994_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a9b00a3d-63fc-466b-b5c8-6668eea5c551_isrForCrosstalkSources_995994_33.stderr
parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: <AppFuture at 0x14991bbe1fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 995e1263c617d9c2eed2190129c082b5
parsl.dataflow.memoization INFO: Task 38 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/4534167a-a22d-4328-95fd-b0968dadb786_isrForCrosstalkSources_995849_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/4534167a-a22d-4328-95fd-b0968dadb786_isrForCrosstalkSources_995849_34.stderr
parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: <AppFuture at 0x14991bbf3310 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 39 has memoization hash d84674fa4bc5915d5a1a89ce6de285aa
parsl.dataflow.memoization INFO: Task 39 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/60767b93-576c-47a9-b842-ac5f21a0bb89_isrForCrosstalkSources_995993_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/60767b93-576c-47a9-b842-ac5f21a0bb89_isrForCrosstalkSources_995993_59.stderr
parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: <AppFuture at 0x14991bbdbdd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 40 has memoization hash d37f0b0def8718f73857ba6876226ccf
parsl.dataflow.memoization INFO: Task 40 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cbfed5d7-4985-4886-8b35-53c163f56493_isrForCrosstalkSources_995994_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cbfed5d7-4985-4886-8b35-53c163f56493_isrForCrosstalkSources_995994_58.stderr
parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: <AppFuture at 0x14991bbda610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 48c9ce2086c524cff7d98711839c6d59
parsl.dataflow.memoization INFO: Task 41 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/9b456bcd-260c-4e96-95d6-3024bc5db428_isrForCrosstalkSources_995849_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/9b456bcd-260c-4e96-95d6-3024bc5db428_isrForCrosstalkSources_995849_32.stderr
parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: <AppFuture at 0x14991bbdabd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 40332e02ac0cd0114f7f3fcb3486516c
parsl.dataflow.memoization INFO: Task 42 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2902b313-45ea-42b8-987e-318fdac5cb36_isrForCrosstalkSources_995995_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2902b313-45ea-42b8-987e-318fdac5cb36_isrForCrosstalkSources_995995_25.stderr
parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: <AppFuture at 0x14991bbcf290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 43 has memoization hash acd9ff658b1c67e180c19c8e208c53d7
parsl.dataflow.memoization INFO: Task 43 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/8888caa2-7ec6-4ae7-9f04-3765fac519ab_isrForCrosstalkSources_995993_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/8888caa2-7ec6-4ae7-9f04-3765fac519ab_isrForCrosstalkSources_995993_16.stderr
parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: <AppFuture at 0x14991bc67d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 1be8191681922ed4ae390b93ffe18452
parsl.dataflow.memoization INFO: Task 44 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/601e645a-2ac4-4066-9fcf-aeb6bad6ddc3_isrForCrosstalkSources_995850_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/601e645a-2ac4-4066-9fcf-aeb6bad6ddc3_isrForCrosstalkSources_995850_15.stderr
parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: <AppFuture at 0x14991bbbfed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 5d74d5d69ed9ec91527a087cea77b95e
parsl.dataflow.memoization INFO: Task 45 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/435afef9-405e-45fc-89b1-2dd748001920_isrForCrosstalkSources_995850_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/435afef9-405e-45fc-89b1-2dd748001920_isrForCrosstalkSources_995850_42.stderr
parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: <AppFuture at 0x14991bbbd090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 34e20fe4bb06ad170ca35472ea5ff7fd
parsl.dataflow.memoization INFO: Task 46 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e07ab3a9-94cc-4fae-95be-b65a8d1c0369_isrForCrosstalkSources_995993_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e07ab3a9-94cc-4fae-95be-b65a8d1c0369_isrForCrosstalkSources_995993_30.stderr
parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: <AppFuture at 0x14991bbbd1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 47 has memoization hash c301fa6e296451cb87ba6b22abbbf4fd
parsl.dataflow.memoization INFO: Task 47 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9379c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1070e4b7-fe62-4d73-ba6b-ba089b794579_isrForCrosstalkSources_995849_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1070e4b7-fe62-4d73-ba6b-ba089b794579_isrForCrosstalkSources_995849_47.stderr
parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: <AppFuture at 0x14991bbb82d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 58d0973fb87d9ed150e321312f060647
parsl.dataflow.memoization INFO: Task 48 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/dfe9d51f-542a-4476-8275-a3164de8bd31_isrForCrosstalkSources_995994_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/dfe9d51f-542a-4476-8275-a3164de8bd31_isrForCrosstalkSources_995994_61.stderr
parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: <AppFuture at 0x14991bbb9090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 49 has memoization hash ad770da9d04d00e0ce31e9f9a5919d8d
parsl.dataflow.memoization INFO: Task 49 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f42c6e01-cf8d-42a3-ae85-3a5848ba63c8_isrForCrosstalkSources_995994_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f42c6e01-cf8d-42a3-ae85-3a5848ba63c8_isrForCrosstalkSources_995994_24.stderr
parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: <AppFuture at 0x14991bbbac90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 50 has memoization hash ad62a31e71cdec5aa336fe9b5ffd3fa5
parsl.dataflow.memoization INFO: Task 50 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/520f2a62-fdc9-4128-a5de-c3984a8e5fcc_isrForCrosstalkSources_995994_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/520f2a62-fdc9-4128-a5de-c3984a8e5fcc_isrForCrosstalkSources_995994_31.stderr
parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: <AppFuture at 0x14991bbadd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 7c2d40167ff8696f8faa643e8bc7b99a
parsl.dataflow.memoization INFO: Task 51 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/2525ca40-0033-4ab0-8c48-f0be11078b9d_isrForCrosstalkSources_995994_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/2525ca40-0033-4ab0-8c48-f0be11078b9d_isrForCrosstalkSources_995994_19.stderr
parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: <AppFuture at 0x14991bbae010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 8b40e110b87f44356be5238ca21b718f
parsl.dataflow.memoization INFO: Task 52 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9376a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dd15f367-c77b-4c89-bc0e-180b3f2a04ad_isrForCrosstalkSources_995848_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dd15f367-c77b-4c89-bc0e-180b3f2a04ad_isrForCrosstalkSources_995848_36.stderr
parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: <AppFuture at 0x14991bbae450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 53 has memoization hash c8be7809df8bc83638e1e7cd6feea86a
parsl.dataflow.memoization INFO: Task 53 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/bda30d32-0e42-4143-bc0e-99df36e4cee7_isrForCrosstalkSources_995848_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/bda30d32-0e42-4143-bc0e-99df36e4cee7_isrForCrosstalkSources_995848_21.stderr
parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: <AppFuture at 0x14991bb9d750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 54 has memoization hash adef4a7aedff9e2cdad9af3c2095acd9
parsl.dataflow.memoization INFO: Task 54 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9374c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/b8a347b1-e6e0-4fbe-9d7a-7934aa4e4327_isrForCrosstalkSources_995993_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/b8a347b1-e6e0-4fbe-9d7a-7934aa4e4327_isrForCrosstalkSources_995993_24.stderr
parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: <AppFuture at 0x14991bb9d510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 9a748d3d27ab01d30445225a2d21873f
parsl.dataflow.memoization INFO: Task 55 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fdb58824-29ce-43d8-94fc-7aa8ab1be24f_isrForCrosstalkSources_995995_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fdb58824-29ce-43d8-94fc-7aa8ab1be24f_isrForCrosstalkSources_995995_26.stderr
parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: <AppFuture at 0x14991bb9f850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 89aead8f53f4ceb69fd37d4b0fb397ca
parsl.dataflow.memoization INFO: Task 56 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0629d4cb-7bd9-497a-a735-108369eeed7e_isrForCrosstalkSources_995848_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0629d4cb-7bd9-497a-a735-108369eeed7e_isrForCrosstalkSources_995848_41.stderr
parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: <AppFuture at 0x14991bb9c450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 18c041828873f05c1ded1676fc1ef49c
parsl.dataflow.memoization INFO: Task 57 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b937380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5c31adce-642f-438d-9c76-de9ea7da24d4_isrForCrosstalkSources_995848_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5c31adce-642f-438d-9c76-de9ea7da24d4_isrForCrosstalkSources_995848_60.stderr
parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: <AppFuture at 0x14991bb94890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 58 has memoization hash bef86267252617049afdfba4bfa7096e
parsl.dataflow.memoization INFO: Task 58 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9365c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/579078d2-6c75-4fc2-8184-231ae85b1b32_isrForCrosstalkSources_995994_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/579078d2-6c75-4fc2-8184-231ae85b1b32_isrForCrosstalkSources_995994_34.stderr
parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: <AppFuture at 0x14991bb96ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 8f24425dad89f2993ca829c19cf40f49
parsl.dataflow.memoization INFO: Task 59 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9371a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/496735c3-2a63-40a2-8e49-d540dd4a5c38_isrForCrosstalkSources_995848_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/496735c3-2a63-40a2-8e49-d540dd4a5c38_isrForCrosstalkSources_995848_17.stderr
parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: <AppFuture at 0x14991bd79d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 0585bb1dc816e7999784cee34a167a1f
parsl.dataflow.memoization INFO: Task 60 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/f53bf352-7675-4cfd-a834-595ebd81413c_isrForCrosstalkSources_995993_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/f53bf352-7675-4cfd-a834-595ebd81413c_isrForCrosstalkSources_995993_27.stderr
parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: <AppFuture at 0x14991bd78fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 43d1c314cda2ff039563d78e48273838
parsl.dataflow.memoization INFO: Task 61 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/056b47f2-5ef0-41a9-a6b6-3235a07cdc19_isrForCrosstalkSources_995850_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/056b47f2-5ef0-41a9-a6b6-3235a07cdc19_isrForCrosstalkSources_995850_14.stderr
parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: <AppFuture at 0x14991bd79bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 363dab088c2d4858380c183d698d09ce
parsl.dataflow.memoization INFO: Task 62 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/7be2f87b-ac8f-4923-a4bd-3d2b334a102a_isrForCrosstalkSources_995993_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/7be2f87b-ac8f-4923-a4bd-3d2b334a102a_isrForCrosstalkSources_995993_11.stderr
parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: <AppFuture at 0x14991bd71a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 8fbf93b714b6c96e7c3f9c1697450677
parsl.dataflow.memoization INFO: Task 63 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/8fc311ba-0c20-4eeb-96aa-064c673361eb_isrForCrosstalkSources_995993_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/8fc311ba-0c20-4eeb-96aa-064c673361eb_isrForCrosstalkSources_995993_14.stderr
parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: <AppFuture at 0x14991bd73e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 64 has memoization hash d12dc86e7deae959cfa59afac4986517
parsl.dataflow.memoization INFO: Task 64 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/28a5e8f6-f738-4c6a-8efb-f41d6169b60d_isrForCrosstalkSources_995993_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/28a5e8f6-f738-4c6a-8efb-f41d6169b60d_isrForCrosstalkSources_995993_61.stderr
parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: <AppFuture at 0x14991bd73ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 9de13cb58e11c889cb9fbfc02a398206
parsl.dataflow.memoization INFO: Task 65 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/d95f0956-b08d-4d7c-ba23-f39fba457d62_isrForCrosstalkSources_995993_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/d95f0956-b08d-4d7c-ba23-f39fba457d62_isrForCrosstalkSources_995993_7.stderr
parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: <AppFuture at 0x14991bd57b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 70fa3efd2733968975201ca419470461
parsl.dataflow.memoization INFO: Task 66 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/e5d8e6e4-e718-4f6b-b6a3-4a57006fddef_isrForCrosstalkSources_995995_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/e5d8e6e4-e718-4f6b-b6a3-4a57006fddef_isrForCrosstalkSources_995995_40.stderr
parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: <AppFuture at 0x14991bd54d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 99c7acd8324d811edcab3f01c9bb9783
parsl.dataflow.memoization INFO: Task 67 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/50bb205a-cb68-44c1-9472-c92ef274b372_isrForCrosstalkSources_995993_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/50bb205a-cb68-44c1-9472-c92ef274b372_isrForCrosstalkSources_995993_9.stderr
parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: <AppFuture at 0x14991bd55290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 68 has memoization hash dd8a0106a5d7fb4f3dea514b70ae3a83
parsl.dataflow.memoization INFO: Task 68 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/4eb269d2-0a46-407b-901b-9e6f178595a3_isrForCrosstalkSources_995993_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/4eb269d2-0a46-407b-901b-9e6f178595a3_isrForCrosstalkSources_995993_22.stderr
parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: <AppFuture at 0x14991bd50050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 69 has memoization hash c7c302fcec884228cd98a03beb05c827
parsl.dataflow.memoization INFO: Task 69 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9745e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dcaf2fa3-7807-43ce-93f7-d8e58ef9f2ed_isrForCrosstalkSources_995848_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dcaf2fa3-7807-43ce-93f7-d8e58ef9f2ed_isrForCrosstalkSources_995848_19.stderr
parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: <AppFuture at 0x14991bd51b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 02843e36656a778dd7ee955b28353555
parsl.dataflow.memoization INFO: Task 70 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cc767d7d-e5d6-49f5-9e76-1e1f3794e1ba_isrForCrosstalkSources_995994_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cc767d7d-e5d6-49f5-9e76-1e1f3794e1ba_isrForCrosstalkSources_995994_29.stderr
parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: <AppFuture at 0x14991bd51d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 6f424550960e3eccd23c88c81a7342f4
parsl.dataflow.memoization INFO: Task 71 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9879411d-d575-4501-a196-8aee0c902967_isrForCrosstalkSources_995850_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9879411d-d575-4501-a196-8aee0c902967_isrForCrosstalkSources_995850_28.stderr
parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: <AppFuture at 0x14991bd52410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 019bb589f7aecb85fe8c2f95d3a716b4
parsl.dataflow.memoization INFO: Task 72 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fc434b24-7c64-4504-a7ef-0e710f4c9938_isrForCrosstalkSources_995995_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fc434b24-7c64-4504-a7ef-0e710f4c9938_isrForCrosstalkSources_995995_30.stderr
parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: <AppFuture at 0x14991bd47d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 73 has memoization hash ad90df79d8dd5003efae99f09271c257
parsl.dataflow.memoization INFO: Task 73 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/f90af71d-2482-491c-b334-9b13d4fd007f_isrForCrosstalkSources_995848_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/f90af71d-2482-491c-b334-9b13d4fd007f_isrForCrosstalkSources_995848_46.stderr
parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: <AppFuture at 0x14991bd44c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 74 has memoization hash f45fcfb82ef4b70f27012ba3160291d7
parsl.dataflow.memoization INFO: Task 74 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/21519b77-8e6e-447a-b516-7d6c89753dc7_isrForCrosstalkSources_995850_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/21519b77-8e6e-447a-b516-7d6c89753dc7_isrForCrosstalkSources_995850_46.stderr
parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: <AppFuture at 0x14991bd45210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 8a6e440374e1c9af7c9dc9347d2e6dcb
parsl.dataflow.memoization INFO: Task 75 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a67207ad-fbe3-4a2f-93c9-9719479e0002_isrForCrosstalkSources_995994_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a67207ad-fbe3-4a2f-93c9-9719479e0002_isrForCrosstalkSources_995994_41.stderr
parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: <AppFuture at 0x14991bd39b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 76 has memoization hash a75a969e9ff788dfa6d7cbd3c11dee5b
parsl.dataflow.memoization INFO: Task 76 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/620fdf74-d2e8-4f4f-8311-629d02b67f17_isrForCrosstalkSources_995995_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/620fdf74-d2e8-4f4f-8311-629d02b67f17_isrForCrosstalkSources_995995_35.stderr
parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: <AppFuture at 0x14991bd38cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 77 has memoization hash a2d9a4bb76b01d64c91a16d3d2bd6260
parsl.dataflow.memoization INFO: Task 77 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/bad4e1ee-3932-4f37-8b86-f0a42867afc9_isrForCrosstalkSources_995994_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/bad4e1ee-3932-4f37-8b86-f0a42867afc9_isrForCrosstalkSources_995994_12.stderr
parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: <AppFuture at 0x14991bd3add0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 6851198a38fb9d443cf28bb4ae5de9f2
parsl.dataflow.memoization INFO: Task 78 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c5705c8f-789c-41a8-b6bd-a53393819441_isrForCrosstalkSources_995848_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c5705c8f-789c-41a8-b6bd-a53393819441_isrForCrosstalkSources_995848_38.stderr
parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: <AppFuture at 0x14991bd47490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 79 has memoization hash b4d331659d72a3018218ac2e81bb984e
parsl.dataflow.memoization INFO: Task 79 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/92302f8c-a532-40e9-9467-d2e4a9b0ea47_isrForCrosstalkSources_995993_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/92302f8c-a532-40e9-9467-d2e4a9b0ea47_isrForCrosstalkSources_995993_12.stderr
parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: <AppFuture at 0x14991bd36c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 2a436ddb9b4e7660f49b5bf1450a0da8
parsl.dataflow.memoization INFO: Task 80 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bfc63bbc-2858-4b2e-887e-e71a37d78016_isrForCrosstalkSources_995850_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bfc63bbc-2858-4b2e-887e-e71a37d78016_isrForCrosstalkSources_995850_43.stderr
parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: <AppFuture at 0x14991bd37090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 81 has memoization hash de88f405776e281f7a0c9734395eccd3
parsl.dataflow.memoization INFO: Task 81 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/710aec35-3bad-4fdc-932e-023c88decabd_isrForCrosstalkSources_995850_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/710aec35-3bad-4fdc-932e-023c88decabd_isrForCrosstalkSources_995850_2.stderr
parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: <AppFuture at 0x14991bd21350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 82 has memoization hash d8bd992230cfab78d1bae8e2dc2281f2
parsl.dataflow.memoization INFO: Task 82 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1c9ba72e-ea4a-4dbc-a191-50a0979fcbd4_isrForCrosstalkSources_995849_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1c9ba72e-ea4a-4dbc-a191-50a0979fcbd4_isrForCrosstalkSources_995849_14.stderr
parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: <AppFuture at 0x14991bd22f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 70db110f67e79bb7126ce27f025b2449
parsl.dataflow.memoization INFO: Task 83 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e3fd52c0-d6b5-4787-a6ad-0a778746554e_isrForCrosstalkSources_995848_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e3fd52c0-d6b5-4787-a6ad-0a778746554e_isrForCrosstalkSources_995848_16.stderr
parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: <AppFuture at 0x14991bd20ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 84 has memoization hash c681e5d2642cab13c4c72cc8150d25e9
parsl.dataflow.memoization INFO: Task 84 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/974621c6-896d-45d9-9864-6ede98b9e2d2_isrForCrosstalkSources_995850_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/974621c6-896d-45d9-9864-6ede98b9e2d2_isrForCrosstalkSources_995850_32.stderr
parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: <AppFuture at 0x14991bd22c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 85 has memoization hash d2e82b6499c96825c37b43e9eb638af6
parsl.dataflow.memoization INFO: Task 85 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/fd909c8a-288e-4a3f-9ab1-0fcec0f40f8f_isrForCrosstalkSources_995850_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/fd909c8a-288e-4a3f-9ab1-0fcec0f40f8f_isrForCrosstalkSources_995850_6.stderr
parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: <AppFuture at 0x14991bd1bcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 432baa5268c6c609e7c4c6f3b5621511
parsl.dataflow.memoization INFO: Task 86 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e497f482-4e5e-41de-99d6-4b836f06ca9f_isrForCrosstalkSources_995994_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e497f482-4e5e-41de-99d6-4b836f06ca9f_isrForCrosstalkSources_995994_18.stderr
parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: <AppFuture at 0x14991bd1b6d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 8004ceee9e44da5fdf3d70b5b89b670b
parsl.dataflow.memoization INFO: Task 87 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9751c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/1f7f0304-04ed-4a86-85b8-361a1f42ad9a_isrForCrosstalkSources_995995_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/1f7f0304-04ed-4a86-85b8-361a1f42ad9a_isrForCrosstalkSources_995995_8.stderr
parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: <AppFuture at 0x14991bd1a550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 326c57cdc5df5582a9109a32078f3676
parsl.dataflow.memoization INFO: Task 88 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2b0a8d19-5776-477f-ad63-f9ce3c602dd9_isrForCrosstalkSources_995995_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2b0a8d19-5776-477f-ad63-f9ce3c602dd9_isrForCrosstalkSources_995995_56.stderr
parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: <AppFuture at 0x14991bd0d790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 89 has memoization hash c5e59026211390ef2f58e863f202c652
parsl.dataflow.memoization INFO: Task 89 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0edde73c-2a95-4e0d-873c-451890054978_isrForCrosstalkSources_995995_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0edde73c-2a95-4e0d-873c-451890054978_isrForCrosstalkSources_995995_5.stderr
parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: <AppFuture at 0x14992c2789d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 80ebd0bd90d7ffe9ab2032589d2c55c0
parsl.dataflow.memoization INFO: Task 90 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9753a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/4beba3cb-e25b-48b4-852f-11854b53c436_isrForCrosstalkSources_995993_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/4beba3cb-e25b-48b4-852f-11854b53c436_isrForCrosstalkSources_995993_48.stderr
parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: <AppFuture at 0x14991bb949d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 0445a99289f77ec5583f15868ec0fc0f
parsl.dataflow.memoization INFO: Task 91 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/313b574c-b1d5-4ec6-872b-33cfdab3cbb3_isrForCrosstalkSources_995994_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/313b574c-b1d5-4ec6-872b-33cfdab3cbb3_isrForCrosstalkSources_995994_15.stderr
parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: <AppFuture at 0x14991bcfda50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 1de8c602c65ed2fb8223054b482ed1c0
parsl.dataflow.memoization INFO: Task 92 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/18389d78-28e5-402e-ba89-efa9df3a6879_isrForCrosstalkSources_995995_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/18389d78-28e5-402e-ba89-efa9df3a6879_isrForCrosstalkSources_995995_15.stderr
parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: <AppFuture at 0x14991bcfc390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 758153486b6c5fe9561682da49cbe1af
parsl.dataflow.memoization INFO: Task 93 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f87eeefc-75d2-47cf-8d67-05bc23d2433f_isrForCrosstalkSources_995995_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f87eeefc-75d2-47cf-8d67-05bc23d2433f_isrForCrosstalkSources_995995_33.stderr
parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: <AppFuture at 0x14991bcfd9d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 8d7126091a19407dc8f0782036d16705
parsl.dataflow.memoization INFO: Task 94 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/b9e15fcf-05d3-44b8-a689-d554f7e97c08_isrForCrosstalkSources_995850_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/b9e15fcf-05d3-44b8-a689-d554f7e97c08_isrForCrosstalkSources_995850_58.stderr
parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: <AppFuture at 0x14991bcff010 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 92e741e3fbf14e92c1729e63b1610237
parsl.dataflow.memoization INFO: Task 95 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9756c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/31070305-650c-465f-a050-f52c0d3846f6_isrForCrosstalkSources_995993_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/31070305-650c-465f-a050-f52c0d3846f6_isrForCrosstalkSources_995993_60.stderr
parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: <AppFuture at 0x14991bcf5e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 32d94d2d14da4255bb683c2fe194adc6
parsl.dataflow.memoization INFO: Task 96 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c7fb1623-51d2-4585-af0b-99aad85a25a1_isrForCrosstalkSources_995850_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c7fb1623-51d2-4585-af0b-99aad85a25a1_isrForCrosstalkSources_995850_36.stderr
parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: <AppFuture at 0x14991bcf5b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 97 has memoization hash e6379ee0c3aaa86a561951a685ec7236
parsl.dataflow.memoization INFO: Task 97 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/1181f88d-f618-4776-aea6-f679edbf3e48_isrForCrosstalkSources_995993_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/1181f88d-f618-4776-aea6-f679edbf3e48_isrForCrosstalkSources_995993_5.stderr
parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: <AppFuture at 0x14991bcf5990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 561951e89ccc748a1e959f34699dae21
parsl.dataflow.memoization INFO: Task 98 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/276376e4-025f-4d6e-bf4f-bc8b4a203689_isrForCrosstalkSources_995994_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/276376e4-025f-4d6e-bf4f-bc8b4a203689_isrForCrosstalkSources_995994_40.stderr
parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: <AppFuture at 0x14991bc0ecd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 1338e920c8513608ee504fd338c7b455
parsl.dataflow.memoization INFO: Task 99 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/ede051e6-f070-4469-885e-6c7ff26c7e8a_isrForCrosstalkSources_995995_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/ede051e6-f070-4469-885e-6c7ff26c7e8a_isrForCrosstalkSources_995995_52.stderr
parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: <AppFuture at 0x14991bcee610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 3bdc31ba8b30bde73190c171894912f0
parsl.dataflow.memoization INFO: Task 100 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/79f4c140-c19e-4b23-a692-d0f95a3bc8d7_isrForCrosstalkSources_995993_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/79f4c140-c19e-4b23-a692-d0f95a3bc8d7_isrForCrosstalkSources_995993_35.stderr
parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: <AppFuture at 0x14991bcef110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 101 has memoization hash a196fb5aa8fcf4ec8c3045b496a3f0ac
parsl.dataflow.memoization INFO: Task 101 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/cb1d3409-4a14-4aec-a4fe-cd5268da7a8d_isrForCrosstalkSources_995848_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/cb1d3409-4a14-4aec-a4fe-cd5268da7a8d_isrForCrosstalkSources_995848_35.stderr
parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: <AppFuture at 0x14991bcee0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 947b15f80d340b24bc01318732b72b03
parsl.dataflow.memoization INFO: Task 102 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a1237ff5-e482-46d0-aabd-b80e9300337c_isrForCrosstalkSources_995848_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a1237ff5-e482-46d0-aabd-b80e9300337c_isrForCrosstalkSources_995848_9.stderr
parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: <AppFuture at 0x14991bce6490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 998a8972ace790be9c0a4166c8ed0155
parsl.dataflow.memoization INFO: Task 103 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/99e3f1f9-780b-469d-800c-a378a5dd0d18_isrForCrosstalkSources_995994_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/99e3f1f9-780b-469d-800c-a378a5dd0d18_isrForCrosstalkSources_995994_35.stderr
parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: <AppFuture at 0x14991bce7f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 6a4e7041a0a1df92b94179c7a590dfc6
parsl.dataflow.memoization INFO: Task 104 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/6b9a7b91-b7b7-42df-9866-6c777094fb17_isrForCrosstalkSources_995848_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/6b9a7b91-b7b7-42df-9866-6c777094fb17_isrForCrosstalkSources_995848_53.stderr
parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: <AppFuture at 0x14991bce4150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 09a1270fb9ba5d47844d90f129f5faeb
parsl.dataflow.memoization INFO: Task 105 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1a946e96-7211-46e9-9c24-6c8f5a4c82e1_isrForCrosstalkSources_995849_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1a946e96-7211-46e9-9c24-6c8f5a4c82e1_isrForCrosstalkSources_995849_50.stderr
parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: <AppFuture at 0x14991bce6ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 106 has memoization hash af1323d6a3f0b224f1d6021f93357fb2
parsl.dataflow.memoization INFO: Task 106 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1691016a-6af0-4f79-a9f9-600aa3e4ae64_isrForCrosstalkSources_995849_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1691016a-6af0-4f79-a9f9-600aa3e4ae64_isrForCrosstalkSources_995849_48.stderr
parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: <AppFuture at 0x14991bce4d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 04bbefb9cd07cf05eff71c96427016e4
parsl.dataflow.memoization INFO: Task 107 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/bf904008-8ecd-47c7-b43e-dcb17cfd53d4_isrForCrosstalkSources_995848_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/bf904008-8ecd-47c7-b43e-dcb17cfd53d4_isrForCrosstalkSources_995848_27.stderr
parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: <AppFuture at 0x14991bcdfad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 416aa34eb032c355a4ad808c6ef31e11
parsl.dataflow.memoization INFO: Task 108 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/1090f174-b2af-43c6-895c-d568509c640e_isrForCrosstalkSources_995995_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/1090f174-b2af-43c6-895c-d568509c640e_isrForCrosstalkSources_995995_4.stderr
parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: <AppFuture at 0x14991bcdf390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 109 has memoization hash df954223226b19c36651cb85a45bc444
parsl.dataflow.memoization INFO: Task 109 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b975ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c55b725f-0b03-4b5c-9833-c2c3859fdf31_isrForCrosstalkSources_995848_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c55b725f-0b03-4b5c-9833-c2c3859fdf31_isrForCrosstalkSources_995848_23.stderr
parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: <AppFuture at 0x14991bcdd410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 110 has memoization hash cb24b5a271c5ed005c9605d0e12daa1c
parsl.dataflow.memoization INFO: Task 110 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/871fd4d0-5cc3-4d4b-96c7-9c8404f72fcc_isrForCrosstalkSources_995994_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/871fd4d0-5cc3-4d4b-96c7-9c8404f72fcc_isrForCrosstalkSources_995994_10.stderr
parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: <AppFuture at 0x14991bcdc1d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 46cb2297eb73d46e2b4e616957f9b098
parsl.dataflow.memoization INFO: Task 111 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9760c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/13b19db7-bf26-4143-8d7b-4fd821cb5829_isrForCrosstalkSources_995993_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/13b19db7-bf26-4143-8d7b-4fd821cb5829_isrForCrosstalkSources_995993_4.stderr
parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: <AppFuture at 0x14991bcd5290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 0718e451e8964d146b36836d248cca48
parsl.dataflow.memoization INFO: Task 112 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c111801c-d93e-4f32-9783-82b1f285508e_isrForCrosstalkSources_995850_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c111801c-d93e-4f32-9783-82b1f285508e_isrForCrosstalkSources_995850_60.stderr
parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: <AppFuture at 0x14991bcd7fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 2b5efd4fc325260b5b6525cc5cbc67e4
parsl.dataflow.memoization INFO: Task 113 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9f070c5e-5a0a-4e47-8d2f-2412b1b1e7f3_isrForCrosstalkSources_995993_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9f070c5e-5a0a-4e47-8d2f-2412b1b1e7f3_isrForCrosstalkSources_995993_56.stderr
parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: <AppFuture at 0x14991bcd7250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 944a3788fb4f43999f739e351f67dd48
parsl.dataflow.memoization INFO: Task 114 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/b0f43500-ed04-484b-bd6f-8191a5d517cf_isrForCrosstalkSources_995993_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/b0f43500-ed04-484b-bd6f-8191a5d517cf_isrForCrosstalkSources_995993_50.stderr
parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: <AppFuture at 0x14991bcd4150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 115 has memoization hash f673ba8c54c5761c0bcbb849c133cf5e
parsl.dataflow.memoization INFO: Task 115 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9d15e8a9-62fa-408b-a849-fa486aeba5df_isrForCrosstalkSources_995995_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9d15e8a9-62fa-408b-a849-fa486aeba5df_isrForCrosstalkSources_995995_27.stderr
parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: <AppFuture at 0x14991bcd5d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 3d70be0d3cb6373e3cb4f50383bc257c
parsl.dataflow.memoization INFO: Task 116 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c7f0d776-e002-4ae8-b756-0a4246c1d2fb_isrForCrosstalkSources_995850_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c7f0d776-e002-4ae8-b756-0a4246c1d2fb_isrForCrosstalkSources_995850_49.stderr
parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: <AppFuture at 0x14991bcb5f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 06e1e04407716d8a4b9b3ce6c275c9a9
parsl.dataflow.memoization INFO: Task 117 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/f98cb30a-cc90-431f-bd20-6976ba452936_isrForCrosstalkSources_995849_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/f98cb30a-cc90-431f-bd20-6976ba452936_isrForCrosstalkSources_995849_43.stderr
parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: <AppFuture at 0x14991bcb4810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 118 has memoization hash c855e7a7a178949d79085c1c00351adf
parsl.dataflow.memoization INFO: Task 118 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/b5a773d5-13d0-48f4-b017-02bd9e7462a2_isrForCrosstalkSources_995994_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/b5a773d5-13d0-48f4-b017-02bd9e7462a2_isrForCrosstalkSources_995994_7.stderr
parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: <AppFuture at 0x14991bcb7cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 624fb369bec5dbec048f8e398c26e7f5
parsl.dataflow.memoization INFO: Task 119 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/ee8ff856-be56-47f5-bd4d-9d133e837291_isrForCrosstalkSources_995850_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/ee8ff856-be56-47f5-bd4d-9d133e837291_isrForCrosstalkSources_995850_10.stderr
parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: <AppFuture at 0x14991bcb4b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 280bd72c21cc9176ccfd3df042190045
parsl.dataflow.memoization INFO: Task 120 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/368a341a-725a-4031-9c59-6a7ea01418e4_isrForCrosstalkSources_995849_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/368a341a-725a-4031-9c59-6a7ea01418e4_isrForCrosstalkSources_995849_61.stderr
parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: <AppFuture at 0x14991bcb61d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 121 has memoization hash fa27600603f15b5c1c55513185cdbc42
parsl.dataflow.memoization INFO: Task 121 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/129df106-3a00-4475-b4f2-93f4ceb61ed4_isrForCrosstalkSources_995850_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/129df106-3a00-4475-b4f2-93f4ceb61ed4_isrForCrosstalkSources_995850_3.stderr
parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: <AppFuture at 0x14991bcce350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 4fb01dc79eec1e2dc7656a1e56044609
parsl.dataflow.memoization INFO: Task 122 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c71c7bc9-d085-4cad-97b8-f625e5f8a4f5_isrForCrosstalkSources_995849_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c71c7bc9-d085-4cad-97b8-f625e5f8a4f5_isrForCrosstalkSources_995849_12.stderr
parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: <AppFuture at 0x14991bcced90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 430100961a82d6457c616f2d3f0320f6
parsl.dataflow.memoization INFO: Task 123 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9768e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0e78f02c-d1d3-4e93-9b8c-a5630659edf7_isrForCrosstalkSources_995995_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0e78f02c-d1d3-4e93-9b8c-a5630659edf7_isrForCrosstalkSources_995995_51.stderr
parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: <AppFuture at 0x14991bcce510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 0e1776abe4683c3b081f6a9761ec3dca
parsl.dataflow.memoization INFO: Task 124 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3bce7f8c-477f-4be9-84e2-42a930fa6342_isrForCrosstalkSources_995995_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3bce7f8c-477f-4be9-84e2-42a930fa6342_isrForCrosstalkSources_995995_34.stderr
parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: <AppFuture at 0x14991bcc0f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 125 has memoization hash e933f6e169cd70c0ed8622f78c041ec1
parsl.dataflow.memoization INFO: Task 125 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/10fc11ac-29ee-4e50-ab84-35f8277d7818_isrForCrosstalkSources_995849_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/10fc11ac-29ee-4e50-ab84-35f8277d7818_isrForCrosstalkSources_995849_36.stderr
parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: <AppFuture at 0x14991bcc01d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 43a19de43c0aa6c858bd84308b9d77e0
parsl.dataflow.memoization INFO: Task 126 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/4639780e-ed06-4a21-bdd8-3e4eafded986_isrForCrosstalkSources_995995_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/4639780e-ed06-4a21-bdd8-3e4eafded986_isrForCrosstalkSources_995995_10.stderr
parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: <AppFuture at 0x14991bcc3c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 127 has memoization hash bc971fbde3dc4c882bcdc6f2a1fb7ad5
parsl.dataflow.memoization INFO: Task 127 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/ef658202-d368-4f67-9a09-a394c42f2f30_isrForCrosstalkSources_995993_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/ef658202-d368-4f67-9a09-a394c42f2f30_isrForCrosstalkSources_995993_52.stderr
parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: <AppFuture at 0x14991bcb9e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 0a5340d249f644171c256a22612b0810
parsl.dataflow.memoization INFO: Task 128 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/031bbd43-8ccf-4ca6-8dbc-a8fadfd8e57c_isrForCrosstalkSources_995994_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/031bbd43-8ccf-4ca6-8dbc-a8fadfd8e57c_isrForCrosstalkSources_995994_59.stderr
parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: <AppFuture at 0x14991bca7750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 6f8dc4dbef39915170fde28de81fddbb
parsl.dataflow.memoization INFO: Task 129 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/77c938e3-21a4-43d3-9ef5-fc058f9d6bf7_isrForCrosstalkSources_995850_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/77c938e3-21a4-43d3-9ef5-fc058f9d6bf7_isrForCrosstalkSources_995850_44.stderr
parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: <AppFuture at 0x149926211250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 130 has memoization hash f824dac351af9c04eb7a9eef1b7d926f
parsl.dataflow.memoization INFO: Task 130 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/12b05366-7f24-4fc9-a6a4-26af10861463_isrForCrosstalkSources_995850_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/12b05366-7f24-4fc9-a6a4-26af10861463_isrForCrosstalkSources_995850_17.stderr
parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: <AppFuture at 0x149926210050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 313e38acba438f3376afb697a9bda262
parsl.dataflow.memoization INFO: Task 131 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a75fa8bb-210f-421f-be20-98f492a1819d_isrForCrosstalkSources_995995_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a75fa8bb-210f-421f-be20-98f492a1819d_isrForCrosstalkSources_995995_62.stderr
parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: <AppFuture at 0x14992619f750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 132 has memoization hash c164db9fc4a6c229e88fe865b9bfaeec
parsl.dataflow.memoization INFO: Task 132 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/73ff802b-76c8-4d64-9f7f-78d8de81d789_isrForCrosstalkSources_995993_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/73ff802b-76c8-4d64-9f7f-78d8de81d789_isrForCrosstalkSources_995993_26.stderr
parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: <AppFuture at 0x14992619eb10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 88d07c3002651f0666c0e1c3f8fea83e
parsl.dataflow.memoization INFO: Task 133 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4e8c55cd-953a-4f83-8f59-63694c2816b3_isrForCrosstalkSources_995994_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4e8c55cd-953a-4f83-8f59-63694c2816b3_isrForCrosstalkSources_995994_2.stderr
parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: <AppFuture at 0x14992619e650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 7159d867575def6f348fd54a8048b5fa
parsl.dataflow.memoization INFO: Task 134 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/293dd85d-7d5d-4d58-abb7-0cd460e87819_isrForCrosstalkSources_995995_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/293dd85d-7d5d-4d58-abb7-0cd460e87819_isrForCrosstalkSources_995995_11.stderr
parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: <AppFuture at 0x14991bc2b8d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 30ee06b530bfc6ed1d94ae29ba3495a0
parsl.dataflow.memoization INFO: Task 135 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b976fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/1c40addb-6103-4efd-a349-893351bfdb07_isrForCrosstalkSources_995848_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/1c40addb-6103-4efd-a349-893351bfdb07_isrForCrosstalkSources_995848_4.stderr
parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: <AppFuture at 0x14991bd81490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 31abcf216c3e94a46a56c029ae28744f
parsl.dataflow.memoization INFO: Task 136 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9cec309e-2bc6-4e9c-a7c6-ea83ec2ca5f4_isrForCrosstalkSources_995850_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9cec309e-2bc6-4e9c-a7c6-ea83ec2ca5f4_isrForCrosstalkSources_995850_37.stderr
parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: <AppFuture at 0x14991bd80b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 10d6c91ebdddc0a720d063e97cf737c4
parsl.dataflow.memoization INFO: Task 137 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/1118cbed-f5da-4a6a-aaf3-995fa5b9d7aa_isrForCrosstalkSources_995850_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/1118cbed-f5da-4a6a-aaf3-995fa5b9d7aa_isrForCrosstalkSources_995850_54.stderr
parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: <AppFuture at 0x14991bd83f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 138 has memoization hash a0c1455e3341d9d504388b7e9195e0b2
parsl.dataflow.memoization INFO: Task 138 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/6781ee82-2327-44b1-8b39-ac983801e564_isrForCrosstalkSources_995995_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/6781ee82-2327-44b1-8b39-ac983801e564_isrForCrosstalkSources_995995_16.stderr
parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: <AppFuture at 0x14991bf66c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 93244afba29ddb05675aabc0b9e17c8a
parsl.dataflow.memoization INFO: Task 139 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9771a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/85555cb0-8fa6-4072-879e-25c6ab65b75d_isrForCrosstalkSources_995993_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/85555cb0-8fa6-4072-879e-25c6ab65b75d_isrForCrosstalkSources_995993_38.stderr
parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: <AppFuture at 0x14992619e990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 927367ea91528c0586a2b372f18581cf
parsl.dataflow.memoization INFO: Task 140 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9772e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/61c36c85-df6f-403f-a71d-1ac094eba8fd_isrForCrosstalkSources_995995_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/61c36c85-df6f-403f-a71d-1ac094eba8fd_isrForCrosstalkSources_995995_48.stderr
parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: <AppFuture at 0x14991ba0c410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 736ad5d017b617be2e0727db3a6ecdb9
parsl.dataflow.memoization INFO: Task 141 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ee12d54a-18e8-4dc9-be79-8bda6f448c8e_isrForCrosstalkSources_995849_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ee12d54a-18e8-4dc9-be79-8bda6f448c8e_isrForCrosstalkSources_995849_60.stderr
parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: <AppFuture at 0x14991ba0d690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 142 has memoization hash a19f5d1c7fb897dc44a4c82a74a652d7
parsl.dataflow.memoization INFO: Task 142 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/3d7c468f-74c6-4aee-9014-4acda0ee8397_isrForCrosstalkSources_995850_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/3d7c468f-74c6-4aee-9014-4acda0ee8397_isrForCrosstalkSources_995850_22.stderr
parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: <AppFuture at 0x14991ba0fd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 143 has memoization hash e53a0631059547f06d596960737dd59e
parsl.dataflow.memoization INFO: Task 143 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/54e1e4c5-77a8-48f1-8100-2abd40d13b7b_isrForCrosstalkSources_995848_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/54e1e4c5-77a8-48f1-8100-2abd40d13b7b_isrForCrosstalkSources_995848_2.stderr
parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: <AppFuture at 0x14991ba0c590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 0ac12ac44899fb17a48e931df1607a7c
parsl.dataflow.memoization INFO: Task 144 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9774c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/34313f25-23c2-4162-80fb-1cc635bb77fd_isrForCrosstalkSources_995849_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/34313f25-23c2-4162-80fb-1cc635bb77fd_isrForCrosstalkSources_995849_11.stderr
parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: <AppFuture at 0x14991b8aa810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 13212acefcbdf4151ce740f8d3699ee2
parsl.dataflow.memoization INFO: Task 145 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/78c08715-db15-4a81-a8c5-6cf058726709_isrForCrosstalkSources_995993_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/78c08715-db15-4a81-a8c5-6cf058726709_isrForCrosstalkSources_995993_23.stderr
parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: <AppFuture at 0x14991b8a9d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 76eb6ddd162a6acbe1d50fb73d18325b
parsl.dataflow.memoization INFO: Task 146 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4b66d040-c6a0-4920-acc2-891468df0220_isrForCrosstalkSources_995994_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4b66d040-c6a0-4920-acc2-891468df0220_isrForCrosstalkSources_995994_8.stderr
parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: <AppFuture at 0x14991b8a93d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 42963926ccc70167f83182fbf076a33f
parsl.dataflow.memoization INFO: Task 147 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/69aff2a1-df2a-4528-8507-b464c07a91ea_isrForCrosstalkSources_995850_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/69aff2a1-df2a-4528-8507-b464c07a91ea_isrForCrosstalkSources_995850_23.stderr
parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: <AppFuture at 0x14991b8abe10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 50199c8fdef787495a43e8a2f8426291
parsl.dataflow.memoization INFO: Task 148 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b85bd10c-3aa5-443a-b1e4-1afb2d259947_isrForCrosstalkSources_995849_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b85bd10c-3aa5-443a-b1e4-1afb2d259947_isrForCrosstalkSources_995849_53.stderr
parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: <AppFuture at 0x14991b8aa090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 149 has memoization hash b59f66db95e59d0414b0dcb84d7214d1
parsl.dataflow.memoization INFO: Task 149 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9777e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/29a5a688-e31b-40f6-8292-96e07869cb90_isrForCrosstalkSources_995995_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/29a5a688-e31b-40f6-8292-96e07869cb90_isrForCrosstalkSources_995995_21.stderr
parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: <AppFuture at 0x14991b9010d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 4afe3e83bb7ea24b691233a8b5fd93f0
parsl.dataflow.memoization INFO: Task 150 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/14498e81-99d9-483e-b480-e5e07842e15e_isrForCrosstalkSources_995995_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/14498e81-99d9-483e-b480-e5e07842e15e_isrForCrosstalkSources_995995_23.stderr
parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: <AppFuture at 0x14991b900250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 0d727b3ff53edb35e2f7b32817766eba
parsl.dataflow.memoization INFO: Task 151 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b9779c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/04a8a54a-a903-42cc-bb01-b6526c1294f6_isrForCrosstalkSources_995848_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/04a8a54a-a903-42cc-bb01-b6526c1294f6_isrForCrosstalkSources_995848_13.stderr
parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: <AppFuture at 0x14991b901bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 2c35cdaed4c8c344e86204718ccdbc51
parsl.dataflow.memoization INFO: Task 152 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3a27aad0-18a5-41fb-b057-65374dadf371_isrForCrosstalkSources_995995_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3a27aad0-18a5-41fb-b057-65374dadf371_isrForCrosstalkSources_995995_38.stderr
parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: <AppFuture at 0x14991b8aad50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 153 has memoization hash f776e768c6a837d1234e95e433a99a40
parsl.dataflow.memoization INFO: Task 153 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/203b08ff-e3d9-4c76-99c5-e28fd3680fa3_isrForCrosstalkSources_995850_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/203b08ff-e3d9-4c76-99c5-e28fd3680fa3_isrForCrosstalkSources_995850_55.stderr
parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: <AppFuture at 0x14991b912b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 154 has memoization hash d3ec9f3e4effce1f510e2d88fe7a371e
parsl.dataflow.memoization INFO: Task 154 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/a80a5b04-305f-49e9-a593-80336c07b3ed_isrForCrosstalkSources_995850_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/a80a5b04-305f-49e9-a593-80336c07b3ed_isrForCrosstalkSources_995850_7.stderr
parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: <AppFuture at 0x14991b910fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 155 has memoization hash a006b23a21e92081ece2b8926fc9db26
parsl.dataflow.memoization INFO: Task 155 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/af31ab4e-eda9-4635-94c9-6e2681225995_isrForCrosstalkSources_995850_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/af31ab4e-eda9-4635-94c9-6e2681225995_isrForCrosstalkSources_995850_45.stderr
parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: <AppFuture at 0x14991b910490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 3b9c5973eb4f4de7f2a19e80cba9780e
parsl.dataflow.memoization INFO: Task 156 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4311957f-f902-49e3-9000-685e81557441_isrForCrosstalkSources_995994_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4311957f-f902-49e3-9000-685e81557441_isrForCrosstalkSources_995994_13.stderr
parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: <AppFuture at 0x14991b911d10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 1398d9f11fbe40adafa65a67d00c28e2
parsl.dataflow.memoization INFO: Task 157 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a563cb65-1797-47b4-a9bb-8acf0df445a0_isrForCrosstalkSources_995848_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a563cb65-1797-47b4-a9bb-8acf0df445a0_isrForCrosstalkSources_995848_24.stderr
parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: <AppFuture at 0x14991b911810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 158 has memoization hash ebf9f07b7568366013786cd0775b3331
parsl.dataflow.memoization INFO: Task 158 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4d806bb5-6f32-4f0e-94b7-e25c8a064aed_isrForCrosstalkSources_995850_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4d806bb5-6f32-4f0e-94b7-e25c8a064aed_isrForCrosstalkSources_995850_62.stderr
parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: <AppFuture at 0x14991a3a08d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 159 has memoization hash b859681ed7270c76f4384aa55543346c
parsl.dataflow.memoization INFO: Task 159 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4c2f1d84-bfe5-4009-8753-2a1b49f54321_isrForCrosstalkSources_995850_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4c2f1d84-bfe5-4009-8753-2a1b49f54321_isrForCrosstalkSources_995850_39.stderr
parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: <AppFuture at 0x14991a3a10d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 160 has memoization hash c30ff55b00b697e12568524237e6b582
parsl.dataflow.memoization INFO: Task 160 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b977f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/01fc0397-284b-4056-8300-9f44a15308ce_isrForCrosstalkSources_995994_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/01fc0397-284b-4056-8300-9f44a15308ce_isrForCrosstalkSources_995994_21.stderr
parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: <AppFuture at 0x14991a3a2710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 161 has memoization hash ff4e23022acfadde31937476c7eb4653
parsl.dataflow.memoization INFO: Task 161 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991b974040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/3bfc88e7-716c-4b4a-9353-daf10f496928_isrForCrosstalkSources_995850_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/3bfc88e7-716c-4b4a-9353-daf10f496928_isrForCrosstalkSources_995850_40.stderr
parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: <AppFuture at 0x14991a3a2b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 026c0e0024d09b3c7a3f36da297b56a0
parsl.dataflow.memoization INFO: Task 162 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2a7d4f7c-38c6-42ee-84d1-10f746eec40f_isrForCrosstalkSources_995848_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2a7d4f7c-38c6-42ee-84d1-10f746eec40f_isrForCrosstalkSources_995848_61.stderr
parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: <AppFuture at 0x14991a3a2d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 163 has memoization hash eed0fd5e11a0555778651e8ac05ebc19
parsl.dataflow.memoization INFO: Task 163 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/76cbffb4-3dfd-4a58-998f-d93e107030ad_isrForCrosstalkSources_995849_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/76cbffb4-3dfd-4a58-998f-d93e107030ad_isrForCrosstalkSources_995849_37.stderr
parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: <AppFuture at 0x14991a3a3490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 164 has memoization hash d20d37f0750d7f3ce755e6726d3a23cd
parsl.dataflow.memoization INFO: Task 164 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a0bd4609-7847-4943-ae95-8b019c6e9161_isrForCrosstalkSources_995995_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a0bd4609-7847-4943-ae95-8b019c6e9161_isrForCrosstalkSources_995995_45.stderr
parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: <AppFuture at 0x14991a3a3a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 587432b979b67c832aee228f293405e2
parsl.dataflow.memoization INFO: Task 165 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/a2dc27cc-4c19-46ff-bc23-e3995e5cbb83_isrForCrosstalkSources_995849_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/a2dc27cc-4c19-46ff-bc23-e3995e5cbb83_isrForCrosstalkSources_995849_39.stderr
parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: <AppFuture at 0x14991a3a0110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 73b560b52e2795ea3a240ab4148bd123
parsl.dataflow.memoization INFO: Task 166 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/5bc31b53-d1d8-41fb-88ca-b5c9740b33b7_isrForCrosstalkSources_995850_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/5bc31b53-d1d8-41fb-88ca-b5c9740b33b7_isrForCrosstalkSources_995850_27.stderr
parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: <AppFuture at 0x14991a3b0f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 40016074939a3ddca3d80f355269bd14
parsl.dataflow.memoization INFO: Task 167 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/35243c84-6dec-44e2-a0d5-ede9d79d4c88_isrForCrosstalkSources_995849_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/35243c84-6dec-44e2-a0d5-ede9d79d4c88_isrForCrosstalkSources_995849_19.stderr
parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: <AppFuture at 0x14991a3b1ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 838c41a4a1a9bacc35f4f0d5a288d38e
parsl.dataflow.memoization INFO: Task 168 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9dfecef5-a6e1-411d-a065-90e1995cf295_isrForCrosstalkSources_995995_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9dfecef5-a6e1-411d-a065-90e1995cf295_isrForCrosstalkSources_995995_18.stderr
parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: <AppFuture at 0x14991a3b2b90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 5a04035c5653c967d8c472db44a01189
parsl.dataflow.memoization INFO: Task 169 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/55af237e-4824-4dc7-84e3-7da75773cd68_isrForCrosstalkSources_995994_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/55af237e-4824-4dc7-84e3-7da75773cd68_isrForCrosstalkSources_995994_38.stderr
parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: <AppFuture at 0x14991a3b1490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 170 has memoization hash d7648629ba65e2c91285f8e3ef2b9d97
parsl.dataflow.memoization INFO: Task 170 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/576844ba-f809-4cae-b46f-48483dfd362e_isrForCrosstalkSources_995993_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/576844ba-f809-4cae-b46f-48483dfd362e_isrForCrosstalkSources_995993_43.stderr
parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: <AppFuture at 0x14991a3b3150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 171 has memoization hash e459ea3863d48ba0d27f167fdb84686d
parsl.dataflow.memoization INFO: Task 171 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/cbc89193-e5b9-42ad-9681-22101ba92622_isrForCrosstalkSources_995995_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/cbc89193-e5b9-42ad-9681-22101ba92622_isrForCrosstalkSources_995995_58.stderr
parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: <AppFuture at 0x14991a3b3f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 39e89a61b7c545e317ce5031cdcf16a1
parsl.dataflow.memoization INFO: Task 172 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e8800f65-5a87-45eb-84a7-780146d87c40_isrForCrosstalkSources_995994_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e8800f65-5a87-45eb-84a7-780146d87c40_isrForCrosstalkSources_995994_53.stderr
parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: <AppFuture at 0x14991a3b3d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 7a62219e8087d6239d756d1fa7293882
parsl.dataflow.memoization INFO: Task 173 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/948f69ac-d1ae-461b-ba60-93b873326ad4_isrForCrosstalkSources_995849_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/948f69ac-d1ae-461b-ba60-93b873326ad4_isrForCrosstalkSources_995849_44.stderr
parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: <AppFuture at 0x14992619ce50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 174 has memoization hash c9e1142462a28b6ed958d433d932ae7d
parsl.dataflow.memoization INFO: Task 174 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e782728f-0cbf-4c20-8cb6-9af35ea53a9a_isrForCrosstalkSources_995849_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e782728f-0cbf-4c20-8cb6-9af35ea53a9a_isrForCrosstalkSources_995849_10.stderr
parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: <AppFuture at 0x14991a3bd690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 8a72fe2afc72904bbe1c59cdd1cd791d
parsl.dataflow.memoization INFO: Task 175 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/0bf6e963-55b9-4784-9e84-550b5b2087b8_isrForCrosstalkSources_995849_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/0bf6e963-55b9-4784-9e84-550b5b2087b8_isrForCrosstalkSources_995849_23.stderr
parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: <AppFuture at 0x14991a3bdd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 176 has memoization hash fe69475aea4bc2d99a33bdb4a250de2b
parsl.dataflow.memoization INFO: Task 176 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/db4d2c48-4195-404e-9f5c-6471d757f934_isrForCrosstalkSources_995850_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/db4d2c48-4195-404e-9f5c-6471d757f934_isrForCrosstalkSources_995850_57.stderr
parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: <AppFuture at 0x14991a3be390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 284011833bb7456b935376ec81ec0967
parsl.dataflow.memoization INFO: Task 177 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d8197fb0-19c1-44fa-a230-c1219dd9b3d4_isrForCrosstalkSources_995850_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d8197fb0-19c1-44fa-a230-c1219dd9b3d4_isrForCrosstalkSources_995850_35.stderr
parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: <AppFuture at 0x14991a3bea90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 178 has memoization hash d798f2eb5010ae0d58b1611e67acb84d
parsl.dataflow.memoization INFO: Task 178 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3acb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/16ca6fb4-dcfb-46e3-ba4c-05ce14a92cbf_isrForCrosstalkSources_995850_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/16ca6fb4-dcfb-46e3-ba4c-05ce14a92cbf_isrForCrosstalkSources_995850_53.stderr
parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: <AppFuture at 0x14991a3bf890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 38aa71ed6600e2f0819f5d0e0d8d1ec8
parsl.dataflow.memoization INFO: Task 179 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3acae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/d79d5d89-e26a-439f-95cf-48b2b408c6b9_isrForCrosstalkSources_995848_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/d79d5d89-e26a-439f-95cf-48b2b408c6b9_isrForCrosstalkSources_995848_44.stderr
parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: <AppFuture at 0x14991a3bee90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 91201b42889f12a843ddfffe895f9e94
parsl.dataflow.memoization INFO: Task 180 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3acc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9e4879cc-8b74-4015-83fa-ec24b1a7331b_isrForCrosstalkSources_995995_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9e4879cc-8b74-4015-83fa-ec24b1a7331b_isrForCrosstalkSources_995995_19.stderr
parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: <AppFuture at 0x14991a3a0a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 181 has memoization hash cb6a4f350a8647e6c5b94024f8a74931
parsl.dataflow.memoization INFO: Task 181 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3accc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ab6cb5fb-332a-4c8a-abb3-06c276aa0770_isrForCrosstalkSources_995994_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ab6cb5fb-332a-4c8a-abb3-06c276aa0770_isrForCrosstalkSources_995994_27.stderr
parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: <AppFuture at 0x14991a3c90d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 1d763fadd87d3064540d383dc4d3eb55
parsl.dataflow.memoization INFO: Task 182 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3acd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e5171b5d-ed60-4d9f-a42f-902dd4bffebe_isrForCrosstalkSources_995849_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e5171b5d-ed60-4d9f-a42f-902dd4bffebe_isrForCrosstalkSources_995849_20.stderr
parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: <AppFuture at 0x14991a3c9110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 5033405f7099523a3bc1b310ece9d5df
parsl.dataflow.memoization INFO: Task 183 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3acea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/f8d5b59e-4b6f-4b53-b34b-215e446d1929_isrForCrosstalkSources_995993_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/f8d5b59e-4b6f-4b53-b34b-215e446d1929_isrForCrosstalkSources_995993_3.stderr
parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: <AppFuture at 0x14991a3c9910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 184 has memoization hash e5047255c14f8b920ad7b37c7de90bfd
parsl.dataflow.memoization INFO: Task 184 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ace00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/9e7f00e0-94a3-454f-b909-44316bf49161_isrForCrosstalkSources_995849_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/9e7f00e0-94a3-454f-b909-44316bf49161_isrForCrosstalkSources_995849_62.stderr
parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: <AppFuture at 0x14991a3c99d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 80ee55f1c194a9d115396238f4c31a3e
parsl.dataflow.memoization INFO: Task 185 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3acf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5cdc8b7a-cafc-405b-81b7-0f014044291e_isrForCrosstalkSources_995848_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5cdc8b7a-cafc-405b-81b7-0f014044291e_isrForCrosstalkSources_995848_40.stderr
parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: <AppFuture at 0x14991a3cafd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 331f254a91267daaa635eec2bf70a0d5
parsl.dataflow.memoization INFO: Task 186 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3acfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/2cce04ed-b191-485d-a7e1-fb40f230191c_isrForCrosstalkSources_995994_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/2cce04ed-b191-485d-a7e1-fb40f230191c_isrForCrosstalkSources_995994_9.stderr
parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: <AppFuture at 0x14991a3caf50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 6490334f1a0a5dd1742398dd27603a11
parsl.dataflow.memoization INFO: Task 187 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/460c2cc6-6ec0-44f8-9414-bf2e6f1d5440_isrForCrosstalkSources_995994_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/460c2cc6-6ec0-44f8-9414-bf2e6f1d5440_isrForCrosstalkSources_995994_52.stderr
parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: <AppFuture at 0x14991a3ca2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 188 has memoization hash e97590027caefaa99bd9ec692aff7a23
parsl.dataflow.memoization INFO: Task 188 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/09726804-eb8c-4c0c-a9b6-ea5719a84bc2_isrForCrosstalkSources_995849_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/09726804-eb8c-4c0c-a9b6-ea5719a84bc2_isrForCrosstalkSources_995849_8.stderr
parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: <AppFuture at 0x14991a3ccd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 3dce88630aff583a171a35075717cc9b
parsl.dataflow.memoization INFO: Task 189 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e10eb275-377f-4fe5-86f1-8e38999474d4_isrForCrosstalkSources_995848_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e10eb275-377f-4fe5-86f1-8e38999474d4_isrForCrosstalkSources_995848_12.stderr
parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: <AppFuture at 0x14991a3cc890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 190 has memoization hash a5dfc40e0b4416bd385f8c5fffab85bc
parsl.dataflow.memoization INFO: Task 190 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5de5e1bb-b02c-4788-ba44-56f9517e0fdf_isrForCrosstalkSources_995848_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5de5e1bb-b02c-4788-ba44-56f9517e0fdf_isrForCrosstalkSources_995848_6.stderr
parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: <AppFuture at 0x14991a3cdd50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 3e81c11cd940c523ee0cb2ee7733e061
parsl.dataflow.memoization INFO: Task 191 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/c7ef614e-ef2c-4a2e-b063-001e4152729d_isrForCrosstalkSources_995995_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/c7ef614e-ef2c-4a2e-b063-001e4152729d_isrForCrosstalkSources_995995_54.stderr
parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: <AppFuture at 0x14991a3ce350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 757c4ac8e96d22245f546cca4527c107
parsl.dataflow.memoization INFO: Task 192 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0deb3db9-733d-4a2e-a6e1-2bfcf9b222cc_isrForCrosstalkSources_995995_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0deb3db9-733d-4a2e-a6e1-2bfcf9b222cc_isrForCrosstalkSources_995995_3.stderr
parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: <AppFuture at 0x14991a3ce850 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 6c1b163121ac0a8dcb0a50a6a4c70bb3
parsl.dataflow.memoization INFO: Task 193 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2b9947ee-b441-4c62-a70c-aafc70cd392d_isrForCrosstalkSources_995849_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2b9947ee-b441-4c62-a70c-aafc70cd392d_isrForCrosstalkSources_995849_33.stderr
parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: <AppFuture at 0x14991a3cf3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 194 has memoization hash cda24867b379ce66fe37d05731be54d7
parsl.dataflow.memoization INFO: Task 194 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4d2a2dbd-c8ca-41d5-8eac-782d0b3d9595_isrForCrosstalkSources_995994_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4d2a2dbd-c8ca-41d5-8eac-782d0b3d9595_isrForCrosstalkSources_995994_25.stderr
parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: <AppFuture at 0x14991a3cf750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 195 has memoization hash cd37f1e2ee84c447ccc4b4ebaa1dbb87
parsl.dataflow.memoization INFO: Task 195 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0f1c9761-4eb4-4359-a3b3-592ec2e4f187_isrForCrosstalkSources_995848_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0f1c9761-4eb4-4359-a3b3-592ec2e4f187_isrForCrosstalkSources_995848_43.stderr
parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: <AppFuture at 0x14991a3cb890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 196 has memoization hash e5a3ee2cddb1612fc0df625bc37b00a7
parsl.dataflow.memoization INFO: Task 196 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f1b53e87-056a-405d-9308-839b5ee0e132_isrForCrosstalkSources_995994_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f1b53e87-056a-405d-9308-839b5ee0e132_isrForCrosstalkSources_995994_60.stderr
parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: <AppFuture at 0x14991a3b12d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 197 has memoization hash a5cba022186cf2e48b3494c529941f53
parsl.dataflow.memoization INFO: Task 197 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/67d1d6a0-c843-45ca-ae94-dd70da994dfa_isrForCrosstalkSources_995993_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/67d1d6a0-c843-45ca-ae94-dd70da994dfa_isrForCrosstalkSources_995993_8.stderr
parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: <AppFuture at 0x14991a3e1d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 4922d3eab87c7bb8c695d3006cbc58dd
parsl.dataflow.memoization INFO: Task 198 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8e72e064-e252-4b4b-9f0b-767e218709ec_isrForCrosstalkSources_995849_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8e72e064-e252-4b4b-9f0b-767e218709ec_isrForCrosstalkSources_995849_1.stderr
parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: <AppFuture at 0x14991a3e2110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 3a7161731aa2bb8c8b12a72982bbb3d0
parsl.dataflow.memoization INFO: Task 199 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/6253e8ca-76f7-4db5-8036-9af7352926a9_isrForCrosstalkSources_995849_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/6253e8ca-76f7-4db5-8036-9af7352926a9_isrForCrosstalkSources_995849_21.stderr
parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: <AppFuture at 0x14991a3e2a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 4c0717d68b836432597131af7c57c2bb
parsl.dataflow.memoization INFO: Task 200 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/b710789a-89e1-4f71-8254-1b9a107960bd_isrForCrosstalkSources_995995_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/b710789a-89e1-4f71-8254-1b9a107960bd_isrForCrosstalkSources_995995_28.stderr
parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: <AppFuture at 0x14991a3e2e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 89d357f9945db447e4bb2953a2dbb3f3
parsl.dataflow.memoization INFO: Task 201 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ec09aa3a-67e3-4819-ab3c-90e94c9a1b6e_isrForCrosstalkSources_995848_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ec09aa3a-67e3-4819-ab3c-90e94c9a1b6e_isrForCrosstalkSources_995848_62.stderr
parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: <AppFuture at 0x14991a3e3c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 5ea3ce2df5d043e7f42b5f944df736e1
parsl.dataflow.memoization INFO: Task 202 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ad9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/cb32a842-3ec8-4bfa-b22e-506a5ded5470_isrForCrosstalkSources_995993_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/cb32a842-3ec8-4bfa-b22e-506a5ded5470_isrForCrosstalkSources_995993_62.stderr
parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: <AppFuture at 0x14991a3e2750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 203 has memoization hash ab2fb84de1218bb01b6d866d456a7d93
parsl.dataflow.memoization INFO: Task 203 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3adb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/5b6eda7e-f323-44ce-aca8-6c106642f5ed_isrForCrosstalkSources_995995_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/5b6eda7e-f323-44ce-aca8-6c106642f5ed_isrForCrosstalkSources_995995_2.stderr
parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: <AppFuture at 0x14991a3e3890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 25ef2d7b3350a3863988778dc344c9f0
parsl.dataflow.memoization INFO: Task 204 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ada80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/4ec43633-4d0f-4132-a766-1447426002d2_isrForCrosstalkSources_995848_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/4ec43633-4d0f-4132-a766-1447426002d2_isrForCrosstalkSources_995848_52.stderr
parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: <AppFuture at 0x14991a3e8cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 0c02f7a69ae020f0894b91b8a6dbd7d3
parsl.dataflow.memoization INFO: Task 205 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3adbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c6c449f4-24e3-4cf7-9bf2-ecc11097f489_isrForCrosstalkSources_995848_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c6c449f4-24e3-4cf7-9bf2-ecc11097f489_isrForCrosstalkSources_995848_30.stderr
parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: <AppFuture at 0x14991a3e9b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 25acf659c91296365238fca524db5024
parsl.dataflow.memoization INFO: Task 206 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3adc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/54abc801-c9a7-4c47-a3d5-4e4fa83e4a2c_isrForCrosstalkSources_995994_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/54abc801-c9a7-4c47-a3d5-4e4fa83e4a2c_isrForCrosstalkSources_995994_11.stderr
parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: <AppFuture at 0x14991a3e9f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 256452ec810ee84f9c64930f35b1a98b
parsl.dataflow.memoization INFO: Task 207 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3add00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ebf381fd-4605-46df-866f-23d19d5b1d48_isrForCrosstalkSources_995849_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ebf381fd-4605-46df-866f-23d19d5b1d48_isrForCrosstalkSources_995849_35.stderr
parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: <AppFuture at 0x14991a3eadd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 208 has memoization hash f07b2df85382484dc93e799e6731de2b
parsl.dataflow.memoization INFO: Task 208 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ade40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/605b71a9-7f69-48ea-917a-26e3bdd5651d_isrForCrosstalkSources_995995_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/605b71a9-7f69-48ea-917a-26e3bdd5651d_isrForCrosstalkSources_995995_7.stderr
parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: <AppFuture at 0x14991a3ea910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 209 has memoization hash fdc322c9d0a450f063711f0ed43af766
parsl.dataflow.memoization INFO: Task 209 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3adda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/d8650899-c629-4e3e-add5-bb96e989da65_isrForCrosstalkSources_995849_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/d8650899-c629-4e3e-add5-bb96e989da65_isrForCrosstalkSources_995849_59.stderr
parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: <AppFuture at 0x14991a3eb290 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 4ccd48078fe182d5348612b3b4cbc1a9
parsl.dataflow.memoization INFO: Task 210 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3adee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/310890b2-49ee-4b26-b70e-b20dd8341355_isrForCrosstalkSources_995848_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/310890b2-49ee-4b26-b70e-b20dd8341355_isrForCrosstalkSources_995848_28.stderr
parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: <AppFuture at 0x14991a3eba50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 61a58fcbda1d8d354489ed2b4efbe2d2
parsl.dataflow.memoization INFO: Task 211 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3adf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/c7709f2f-3d37-4e44-a47a-52a90e9c6244_isrForCrosstalkSources_995994_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/c7709f2f-3d37-4e44-a47a-52a90e9c6244_isrForCrosstalkSources_995994_17.stderr
parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: <AppFuture at 0x14991a3f0a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 212 has memoization hash dbc845d9a924fb0477564bac5e12eecb
parsl.dataflow.memoization INFO: Task 212 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/7792943f-5050-4394-ab8b-e293b47e8fd7_isrForCrosstalkSources_995994_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/7792943f-5050-4394-ab8b-e293b47e8fd7_isrForCrosstalkSources_995994_30.stderr
parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: <AppFuture at 0x14991a3f0a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 33955a2ddc1e8028295f141b0dc62b0f
parsl.dataflow.memoization INFO: Task 213 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/ec671436-3995-40fd-a69a-3e1d2da96c74_isrForCrosstalkSources_995993_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/ec671436-3995-40fd-a69a-3e1d2da96c74_isrForCrosstalkSources_995993_25.stderr
parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: <AppFuture at 0x14991a3f1d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 0333e6b694d6ed3862d8e58c0ae1e026
parsl.dataflow.memoization INFO: Task 214 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a6b206e9-5210-4ffc-9a34-3cf746b98c8e_isrForCrosstalkSources_995848_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a6b206e9-5210-4ffc-9a34-3cf746b98c8e_isrForCrosstalkSources_995848_15.stderr
parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: <AppFuture at 0x14991a3f1bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 0eb5321908c8a441bc5317fb70d8624b
parsl.dataflow.memoization INFO: Task 215 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/08851989-460e-4893-bb6b-84f310a63006_isrForCrosstalkSources_995993_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/08851989-460e-4893-bb6b-84f310a63006_isrForCrosstalkSources_995993_1.stderr
parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: <AppFuture at 0x14991a3f3a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 7aba1b862762299c5665b47a02c80beb
parsl.dataflow.memoization INFO: Task 216 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/ac71fba8-eb7a-4b17-8676-c2dd13422bca_isrForCrosstalkSources_995995_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/ac71fba8-eb7a-4b17-8676-c2dd13422bca_isrForCrosstalkSources_995995_53.stderr
parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: <AppFuture at 0x14991a3f3050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 58a9926ebb62597b6c5aed4838bb7ade
parsl.dataflow.memoization INFO: Task 217 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4899b1cf-207a-4382-9665-5a11b4f2988f_isrForCrosstalkSources_995850_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4899b1cf-207a-4382-9665-5a11b4f2988f_isrForCrosstalkSources_995850_8.stderr
parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: <AppFuture at 0x14991a3f3ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 989cd4348aa253b90f04369eed42b8f4
parsl.dataflow.memoization INFO: Task 218 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cb28a63a-aeb9-4fe9-90c9-869dc409d536_isrForCrosstalkSources_995994_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cb28a63a-aeb9-4fe9-90c9-869dc409d536_isrForCrosstalkSources_995994_36.stderr
parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: <AppFuture at 0x14991a3f8a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 63228f76f8e67e7d43ef70fd9bab8ab6
parsl.dataflow.memoization INFO: Task 219 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/870d8c29-c66c-46ed-8f84-784fe7d2b52b_isrForCrosstalkSources_995848_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/870d8c29-c66c-46ed-8f84-784fe7d2b52b_isrForCrosstalkSources_995848_10.stderr
parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: <AppFuture at 0x14991a3f91d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 89db8f470efc622c0c90f7cc5c01ef22
parsl.dataflow.memoization INFO: Task 220 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d3c600ea-a05a-4a29-bcfd-9d541163e33b_isrForCrosstalkSources_995850_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d3c600ea-a05a-4a29-bcfd-9d541163e33b_isrForCrosstalkSources_995850_4.stderr
parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: <AppFuture at 0x14991a3f9610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 072d229ada02b292b1ab55abb62907f2
parsl.dataflow.memoization INFO: Task 221 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8ad84c60-0025-4607-80be-1537336fec6a_isrForCrosstalkSources_995850_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8ad84c60-0025-4607-80be-1537336fec6a_isrForCrosstalkSources_995850_9.stderr
parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: <AppFuture at 0x14991a3fabd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 222 has memoization hash d41b10e276febbbd2aea7e231a73882c
parsl.dataflow.memoization INFO: Task 222 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/3d57a71d-4373-465f-bec7-10cefc4bf9bb_isrForCrosstalkSources_995993_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/3d57a71d-4373-465f-bec7-10cefc4bf9bb_isrForCrosstalkSources_995993_37.stderr
parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: <AppFuture at 0x14991a3fb5d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 9a0923fa427e56be4766a7d0f7cdf891
parsl.dataflow.memoization INFO: Task 223 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/234a3a1d-7478-437c-b3bc-0c46ee8330f0_isrForCrosstalkSources_995848_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/234a3a1d-7478-437c-b3bc-0c46ee8330f0_isrForCrosstalkSources_995848_39.stderr
parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: <AppFuture at 0x14991a3fb110 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 323667d478882400095b1757fcb57b3e
parsl.dataflow.memoization INFO: Task 224 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/465138b3-e084-457d-8ebf-976f04cf00e2_isrForCrosstalkSources_995993_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/465138b3-e084-457d-8ebf-976f04cf00e2_isrForCrosstalkSources_995993_2.stderr
parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: <AppFuture at 0x14991a3fbb50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 225 has memoization hash bb4b35a0f12fc518d6ffe1bc77f10c65
parsl.dataflow.memoization INFO: Task 225 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9ac42e83-9152-49d9-83d6-1f6d5fbf8df9_isrForCrosstalkSources_995993_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9ac42e83-9152-49d9-83d6-1f6d5fbf8df9_isrForCrosstalkSources_995993_13.stderr
parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: <AppFuture at 0x14991a3fbe10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 226 has memoization hash f212debadbdde6bb0889f12cf517126f
parsl.dataflow.memoization INFO: Task 226 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/5f7794b3-2824-4712-b5ea-124e7881c43f_isrForCrosstalkSources_995849_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/5f7794b3-2824-4712-b5ea-124e7881c43f_isrForCrosstalkSources_995849_18.stderr
parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: <AppFuture at 0x14991a408a10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 227 has memoization hash ba10774e2bc37b97f1e6ea5ba26d74c7
parsl.dataflow.memoization INFO: Task 227 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ae980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5b18fad0-1fc1-4f33-88fb-0537dff5b33f_isrForCrosstalkSources_995848_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5b18fad0-1fc1-4f33-88fb-0537dff5b33f_isrForCrosstalkSources_995848_37.stderr
parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: <AppFuture at 0x14991a409c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 228 has memoization hash e9f4a068cb31055c398780249f14972b
parsl.dataflow.memoization INFO: Task 228 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c020f498-99eb-42b5-914a-f5d28fa3071c_isrForCrosstalkSources_995849_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c020f498-99eb-42b5-914a-f5d28fa3071c_isrForCrosstalkSources_995849_31.stderr
parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: <AppFuture at 0x14991a40acd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 36a81e9f9ca94deb9b33b0be1918a327
parsl.dataflow.memoization INFO: Task 229 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/3659651f-e170-4c3a-aeed-4736992c9321_isrForCrosstalkSources_995994_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/3659651f-e170-4c3a-aeed-4736992c9321_isrForCrosstalkSources_995994_54.stderr
parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: <AppFuture at 0x14991a40ae90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 5d2d3d905f7d54715b2e8d5a845c4678
parsl.dataflow.memoization INFO: Task 230 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/52d34415-4743-4a47-9954-cf0df8e402b4_isrForCrosstalkSources_995850_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/52d34415-4743-4a47-9954-cf0df8e402b4_isrForCrosstalkSources_995850_21.stderr
parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: <AppFuture at 0x14991a40aed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 231 has memoization hash f746df3be05ea1a2ae8703b0cf3eaeae
parsl.dataflow.memoization INFO: Task 231 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6b0d42fd-a500-493d-8405-6052f6831302_isrForCrosstalkSources_995993_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6b0d42fd-a500-493d-8405-6052f6831302_isrForCrosstalkSources_995993_39.stderr
parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: <AppFuture at 0x14991a40b810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 89db020f5b170f50839d538aedb0cadb
parsl.dataflow.memoization INFO: Task 232 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f481c886-4b79-42fc-83c0-420d24407ce8_isrForCrosstalkSources_995995_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f481c886-4b79-42fc-83c0-420d24407ce8_isrForCrosstalkSources_995995_6.stderr
parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: <AppFuture at 0x14991a40ba90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 3c986fdd4ce9bc0b951047617fd9faea
parsl.dataflow.memoization INFO: Task 233 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a3b2659b-e1a2-44a1-9119-ce9366768fdf_isrForCrosstalkSources_995848_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a3b2659b-e1a2-44a1-9119-ce9366768fdf_isrForCrosstalkSources_995848_34.stderr
parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: <AppFuture at 0x14991a411050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 9ee1d59e229622c05feb5db23a39da59
parsl.dataflow.memoization INFO: Task 234 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e3532a13-d6f7-4d0c-892c-259c9f18947f_isrForCrosstalkSources_995993_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e3532a13-d6f7-4d0c-892c-259c9f18947f_isrForCrosstalkSources_995993_40.stderr
parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: <AppFuture at 0x14991a410b10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 235 has memoization hash ba8f9affc63608da700392da6c08794e
parsl.dataflow.memoization INFO: Task 235 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/744d950e-cc8c-427a-9d7c-7c78189d9931_isrForCrosstalkSources_995994_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/744d950e-cc8c-427a-9d7c-7c78189d9931_isrForCrosstalkSources_995994_1.stderr
parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: <AppFuture at 0x14991a412710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 236 has memoization hash e9891956f226d6f8cf1a9c31bbb66136
parsl.dataflow.memoization INFO: Task 236 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8f254323-c096-4408-b996-5d06a5a23cbb_isrForCrosstalkSources_995849_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8f254323-c096-4408-b996-5d06a5a23cbb_isrForCrosstalkSources_995849_45.stderr
parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: <AppFuture at 0x14991a411250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 23a9c512b7bf18835990a4916384450c
parsl.dataflow.memoization INFO: Task 237 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/243cba2f-6c57-4a3b-a469-dd8104d5f340_isrForCrosstalkSources_995849_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/243cba2f-6c57-4a3b-a469-dd8104d5f340_isrForCrosstalkSources_995849_40.stderr
parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: <AppFuture at 0x14991a413550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 762775f2291dff0e69f05a1702e177c7
parsl.dataflow.memoization INFO: Task 238 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/a7786c6d-637a-49d8-952c-370325f36647_isrForCrosstalkSources_995993_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/a7786c6d-637a-49d8-952c-370325f36647_isrForCrosstalkSources_995993_53.stderr
parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: <AppFuture at 0x14991a413790 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 239 has memoization hash bbbd3bb20fb2887283dc7cc03fd31db3
parsl.dataflow.memoization INFO: Task 239 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/239eb505-8d47-477b-962d-3f4db2f91951_isrForCrosstalkSources_995849_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/239eb505-8d47-477b-962d-3f4db2f91951_isrForCrosstalkSources_995849_42.stderr
parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: <AppFuture at 0x14991a413a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 5b9ac5a3f56117df124a1ea09bd4e1f6
parsl.dataflow.memoization INFO: Task 240 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ed07c81f-ea3c-401b-bed6-ee5db21fa41e_isrForCrosstalkSources_995994_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ed07c81f-ea3c-401b-bed6-ee5db21fa41e_isrForCrosstalkSources_995994_50.stderr
parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: <AppFuture at 0x14991a412090 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 241 has memoization hash d46bb39edb9d7a0eae7a75e16321b7e1
parsl.dataflow.memoization INFO: Task 241 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/011feedf-41b9-4b81-8787-26f35696212e_isrForCrosstalkSources_995993_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/011feedf-41b9-4b81-8787-26f35696212e_isrForCrosstalkSources_995993_32.stderr
parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: <AppFuture at 0x14991a414c10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 24904d974d17380082c232e15cb688ce
parsl.dataflow.memoization INFO: Task 242 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d5fe47d3-9435-4f60-be8c-00921a22ba09_isrForCrosstalkSources_995850_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d5fe47d3-9435-4f60-be8c-00921a22ba09_isrForCrosstalkSources_995850_13.stderr
parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: <AppFuture at 0x14991a415990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 5813111deace37c70b5f860c513caf10
parsl.dataflow.memoization INFO: Task 243 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b11daaaa-6e74-49e0-8bf5-b2747d121e14_isrForCrosstalkSources_995849_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b11daaaa-6e74-49e0-8bf5-b2747d121e14_isrForCrosstalkSources_995849_6.stderr
parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: <AppFuture at 0x14991a415d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 0fe1502b73db89abf1d55e312d7519e7
parsl.dataflow.memoization INFO: Task 244 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/db77a0fc-3b2f-4f02-803f-03f10f733501_isrForCrosstalkSources_995994_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/db77a0fc-3b2f-4f02-803f-03f10f733501_isrForCrosstalkSources_995994_26.stderr
parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: <AppFuture at 0x14991a417350 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 245 has memoization hash f2081b87a18b862ca652948785a649cc
parsl.dataflow.memoization INFO: Task 245 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/6660b4c4-6975-464b-a87c-472a856f42d9_isrForCrosstalkSources_995849_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/6660b4c4-6975-464b-a87c-472a856f42d9_isrForCrosstalkSources_995849_5.stderr
parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: <AppFuture at 0x14991a417d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 246 has memoization hash d62d19c3216504475342ba584d55a36f
parsl.dataflow.memoization INFO: Task 246 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/028036fc-d90f-43af-b8e3-094c2269d7d8_isrForCrosstalkSources_995849_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/028036fc-d90f-43af-b8e3-094c2269d7d8_isrForCrosstalkSources_995849_38.stderr
parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: <AppFuture at 0x14991a417610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 311bbe79b2cc81f6b49fa406889aa830
parsl.dataflow.memoization INFO: Task 247 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/0ad2f158-3fff-4117-bce7-87c0244b3c9c_isrForCrosstalkSources_995850_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/0ad2f158-3fff-4117-bce7-87c0244b3c9c_isrForCrosstalkSources_995850_26.stderr
parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: <AppFuture at 0x14991a414d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 85732cd9e1e127890ced450dc936fd9e
parsl.dataflow.memoization INFO: Task 248 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8dfaed11-9b50-48e4-83bf-f7bf9fafee16_isrForCrosstalkSources_995850_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8dfaed11-9b50-48e4-83bf-f7bf9fafee16_isrForCrosstalkSources_995850_5.stderr
parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: <AppFuture at 0x14991a428510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 10330a3f2b36283752f8191fd5f95757
parsl.dataflow.memoization INFO: Task 249 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/9340fe65-0f5b-48ad-85ba-f878c7647fd1_isrForCrosstalkSources_995994_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/9340fe65-0f5b-48ad-85ba-f878c7647fd1_isrForCrosstalkSources_995994_48.stderr
parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: <AppFuture at 0x14991a4289d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 250 has memoization hash ed945e3ee2eba9ec33448d87e1dce692
parsl.dataflow.memoization INFO: Task 250 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c57b43cb-9e30-4594-9e67-5781f564a08d_isrForCrosstalkSources_995848_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c57b43cb-9e30-4594-9e67-5781f564a08d_isrForCrosstalkSources_995848_20.stderr
parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: <AppFuture at 0x14991a429d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 718afab05378b7c209df3de69d975af3
parsl.dataflow.memoization INFO: Task 251 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ca165be4-2355-4eaf-906e-f08541f227b6_isrForCrosstalkSources_995848_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ca165be4-2355-4eaf-906e-f08541f227b6_isrForCrosstalkSources_995848_33.stderr
parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: <AppFuture at 0x14991a42a550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 623e8bed796ac479f241ce67bdf41e30
parsl.dataflow.memoization INFO: Task 252 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/62c87c26-4905-4152-aa1a-607e3b6f2abd_isrForCrosstalkSources_995848_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/62c87c26-4905-4152-aa1a-607e3b6f2abd_isrForCrosstalkSources_995848_47.stderr
parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: <AppFuture at 0x14991a42bb90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 253 has memoization hash db4079ce6ef152e395142b56661170b7
parsl.dataflow.memoization INFO: Task 253 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3afa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/f1dd7629-4123-4363-b823-164966707a16_isrForCrosstalkSources_995849_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/f1dd7629-4123-4363-b823-164966707a16_isrForCrosstalkSources_995849_24.stderr
parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: <AppFuture at 0x14991a42b450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 51173600ee53321d7e4370d3bcdedb95
parsl.dataflow.memoization INFO: Task 254 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3af9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e21df543-19d9-4865-9447-9cb4a0236a17_isrForCrosstalkSources_995993_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e21df543-19d9-4865-9447-9cb4a0236a17_isrForCrosstalkSources_995993_15.stderr
parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: <AppFuture at 0x14991a42bed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 40f7182b00f0714d931014247e8a367a
parsl.dataflow.memoization INFO: Task 255 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3afb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/75552959-7bfe-4228-a1df-ded6ab3ed41f_isrForCrosstalkSources_995848_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/75552959-7bfe-4228-a1df-ded6ab3ed41f_isrForCrosstalkSources_995848_22.stderr
parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: <AppFuture at 0x14991a42be90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 256 has memoization hash c9fdf741ea4b5de0873d620ff91d5b39
parsl.dataflow.memoization INFO: Task 256 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3afba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/84b57c14-6127-4750-a79e-49cddfcd0ae0_isrForCrosstalkSources_995848_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/84b57c14-6127-4750-a79e-49cddfcd0ae0_isrForCrosstalkSources_995848_49.stderr
parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: <AppFuture at 0x14991a431810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 5946903fdc416498cceaf4f26c8a7ef2
parsl.dataflow.memoization INFO: Task 257 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3afc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2adffdf3-ea8d-4ca9-b60b-ca3e9910f835_isrForCrosstalkSources_995848_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2adffdf3-ea8d-4ca9-b60b-ca3e9910f835_isrForCrosstalkSources_995848_42.stderr
parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: <AppFuture at 0x14991a430390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 258 has memoization hash c141d75598285031a247c97401f7517a
parsl.dataflow.memoization INFO: Task 258 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3afd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/677ef656-729c-4ddb-a752-c901138f06a0_isrForCrosstalkSources_995850_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/677ef656-729c-4ddb-a752-c901138f06a0_isrForCrosstalkSources_995850_24.stderr
parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: <AppFuture at 0x14991a432450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 4f52c77e52f498454b0a190411f0170a
parsl.dataflow.memoization INFO: Task 259 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3afce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/7645d4d6-eb9b-4c99-8b0b-0bedea2028b8_isrForCrosstalkSources_995849_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/7645d4d6-eb9b-4c99-8b0b-0bedea2028b8_isrForCrosstalkSources_995849_30.stderr
parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: <AppFuture at 0x14991a432710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 260 has memoization hash df629c9a0b9bc25430faf98bfa0d1546
parsl.dataflow.memoization INFO: Task 260 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3afe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/57aa5aae-f2e1-46eb-a4c0-a0e08f138e40_isrForCrosstalkSources_995848_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/57aa5aae-f2e1-46eb-a4c0-a0e08f138e40_isrForCrosstalkSources_995848_5.stderr
parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: <AppFuture at 0x14991a4330d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 261 has memoization hash c3e7b8a37b18b5bc23a1197d7dc74cc5
parsl.dataflow.memoization INFO: Task 261 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3afec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/aec48142-fb39-4b65-a506-72810d3eda5e_isrForCrosstalkSources_995993_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/aec48142-fb39-4b65-a506-72810d3eda5e_isrForCrosstalkSources_995993_17.stderr
parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: <AppFuture at 0x14991a433f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 05812054145f2576b51912e8aa3bfcf7
parsl.dataflow.memoization INFO: Task 262 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3aff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/7a79c12b-4310-4960-9894-3dd2aaa79a95_isrForCrosstalkSources_995850_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/7a79c12b-4310-4960-9894-3dd2aaa79a95_isrForCrosstalkSources_995850_30.stderr
parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: <AppFuture at 0x14991a433a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 94b2cfe1e9f2ebf16e6bb715e555e015
parsl.dataflow.memoization INFO: Task 263 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a3ac040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/51dc6a86-cb6b-4a83-ad7d-26d8163ed188_isrForCrosstalkSources_995849_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/51dc6a86-cb6b-4a83-ad7d-26d8163ed188_isrForCrosstalkSources_995849_58.stderr
parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: <AppFuture at 0x14991a431e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 264 has memoization hash e1b55ca66d832def134335cd920fd745
parsl.dataflow.memoization INFO: Task 264 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/6ba23c3f-fc59-40eb-bf41-f70898334e46_isrForCrosstalkSources_995850_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/6ba23c3f-fc59-40eb-bf41-f70898334e46_isrForCrosstalkSources_995850_33.stderr
parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: <AppFuture at 0x14991bcd59d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 47f52dd1b03ffdda7fb122cda5b5158f
parsl.dataflow.memoization INFO: Task 265 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/13018970-48af-4bc0-924c-e3de3d9f07b8_isrForCrosstalkSources_995850_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/13018970-48af-4bc0-924c-e3de3d9f07b8_isrForCrosstalkSources_995850_18.stderr
parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: <AppFuture at 0x14991a416610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 13eba9fb520cf45cb0817308e0a0cabb
parsl.dataflow.memoization INFO: Task 266 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5ab11120-89b7-4aa6-84be-809fef29607b_isrForCrosstalkSources_995994_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5ab11120-89b7-4aa6-84be-809fef29607b_isrForCrosstalkSources_995994_22.stderr
parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: <AppFuture at 0x14991a414d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 4faeb6a032d773e93d316406504ebd69
parsl.dataflow.memoization INFO: Task 267 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4382c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dbd14417-6d56-4d75-a4c7-b472e50919e7_isrForCrosstalkSources_995848_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dbd14417-6d56-4d75-a4c7-b472e50919e7_isrForCrosstalkSources_995848_57.stderr
parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: <AppFuture at 0x14991a43ebd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 917c9020890c5b619124d95f2b0a680b
parsl.dataflow.memoization INFO: Task 268 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/80700b4d-1666-4264-83fd-18ec3ca68289_isrForCrosstalkSources_995993_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/80700b4d-1666-4264-83fd-18ec3ca68289_isrForCrosstalkSources_995993_58.stderr
parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: <AppFuture at 0x14991a43e210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 8cbd0342829824b6bb619fa959d29dfd
parsl.dataflow.memoization INFO: Task 269 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e3d2365f-71ca-4592-bd43-0671932ec901_isrForCrosstalkSources_995849_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e3d2365f-71ca-4592-bd43-0671932ec901_isrForCrosstalkSources_995849_17.stderr
parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: <AppFuture at 0x14991a43fc50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 10b342beac1d1433e17b71c51f513c67
parsl.dataflow.memoization INFO: Task 270 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4384a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/170001b8-1fd0-40fe-b76c-37cd06138dd4_isrForCrosstalkSources_995849_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/170001b8-1fd0-40fe-b76c-37cd06138dd4_isrForCrosstalkSources_995849_25.stderr
parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: <AppFuture at 0x14991a43fd10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 4bb204f1a93faeee7ef37c1625ea6ae5
parsl.dataflow.memoization INFO: Task 271 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/f8d9f080-12dc-4bce-be8d-c858a37d271d_isrForCrosstalkSources_995848_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/f8d9f080-12dc-4bce-be8d-c858a37d271d_isrForCrosstalkSources_995848_32.stderr
parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: <AppFuture at 0x14991a445890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 5cca11204331925db6e222559cda6cf8
parsl.dataflow.memoization INFO: Task 272 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4385e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/846bfe47-17d6-4a86-a0fa-69242dd63180_isrForCrosstalkSources_995993_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/846bfe47-17d6-4a86-a0fa-69242dd63180_isrForCrosstalkSources_995993_54.stderr
parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: <AppFuture at 0x14991a4445d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 78bd6dd620c9ac59d31c1d46cdd9d5b8
parsl.dataflow.memoization INFO: Task 273 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/23a9a687-760c-4628-96c1-b8a7dc4c4c8f_isrForCrosstalkSources_995850_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/23a9a687-760c-4628-96c1-b8a7dc4c4c8f_isrForCrosstalkSources_995850_29.stderr
parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: <AppFuture at 0x14991a445690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 0b34624b9c57b196f7c4bc99532e9778
parsl.dataflow.memoization INFO: Task 274 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2d105d32-b39c-4124-a700-76c4e3dee4ad_isrForCrosstalkSources_995849_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2d105d32-b39c-4124-a700-76c4e3dee4ad_isrForCrosstalkSources_995849_51.stderr
parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: <AppFuture at 0x14991a446450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 6b90e60cbad4f6c27505094fb6a91923
parsl.dataflow.memoization INFO: Task 275 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4387c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/56907ed8-22dc-44c9-a341-a9c143a92f7c_isrForCrosstalkSources_995848_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/56907ed8-22dc-44c9-a341-a9c143a92f7c_isrForCrosstalkSources_995848_29.stderr
parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: <AppFuture at 0x14991a446c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 9462425360779d85a6ce775a39af1172
parsl.dataflow.memoization INFO: Task 276 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8cb2490f-a62b-4614-9892-8f5c4daf3d18_isrForCrosstalkSources_995849_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8cb2490f-a62b-4614-9892-8f5c4daf3d18_isrForCrosstalkSources_995849_3.stderr
parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: <AppFuture at 0x14991a447d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 43bb8ae330f3d0fea1b5356c37bf18d4
parsl.dataflow.memoization INFO: Task 277 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/385851eb-110c-41ab-bef1-6edd42046876_isrForCrosstalkSources_995850_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/385851eb-110c-41ab-bef1-6edd42046876_isrForCrosstalkSources_995850_50.stderr
parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: <AppFuture at 0x14991a447190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 278 has memoization hash a72a316e4daaf2acf6c0ba14a170a6f6
parsl.dataflow.memoization INFO: Task 278 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/21f2db4e-d901-4766-9942-5787e4cdf2a8_isrForCrosstalkSources_995995_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/21f2db4e-d901-4766-9942-5787e4cdf2a8_isrForCrosstalkSources_995995_29.stderr
parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: <AppFuture at 0x14991a454c90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 53361e152da5e7106d97b66b6a2b48b1
parsl.dataflow.memoization INFO: Task 279 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4389a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b71a53cd-270d-4323-8f04-2788c03aaca0_isrForCrosstalkSources_995849_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b71a53cd-270d-4323-8f04-2788c03aaca0_isrForCrosstalkSources_995849_16.stderr
parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: <AppFuture at 0x14991a454910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 73d9332620b4072d3b26bf052fa48a2c
parsl.dataflow.memoization INFO: Task 280 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/507bef1e-a4f0-4b8a-87a8-f33d8adda184_isrForCrosstalkSources_995994_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/507bef1e-a4f0-4b8a-87a8-f33d8adda184_isrForCrosstalkSources_995994_32.stderr
parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: <AppFuture at 0x14991a455ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 33965be499e8cad9e10c295cf97b47f3
parsl.dataflow.memoization INFO: Task 281 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/189b01fa-aa66-4d13-b34a-1038ec230d57_isrForCrosstalkSources_995995_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/189b01fa-aa66-4d13-b34a-1038ec230d57_isrForCrosstalkSources_995995_59.stderr
parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: <AppFuture at 0x14991a456410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 28609ce5cec0fe122022cdd08f76823a
parsl.dataflow.memoization INFO: Task 282 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/217f690d-228a-4a98-9b92-1e89ee67d133_isrForCrosstalkSources_995848_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/217f690d-228a-4a98-9b92-1e89ee67d133_isrForCrosstalkSources_995848_8.stderr
parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: <AppFuture at 0x14991a456e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 0dbce6f2612b17ccc16070cd295a9325
parsl.dataflow.memoization INFO: Task 283 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9d90ecc8-5ab7-460d-a65f-6b813a9af576_isrForCrosstalkSources_995995_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9d90ecc8-5ab7-460d-a65f-6b813a9af576_isrForCrosstalkSources_995995_47.stderr
parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: <AppFuture at 0x14991a456f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 1dff35147d2172fac86b5502af60e317
parsl.dataflow.memoization INFO: Task 284 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2fdc780f-23b9-44cf-83e6-2f90a72d8f86_isrForCrosstalkSources_995848_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2fdc780f-23b9-44cf-83e6-2f90a72d8f86_isrForCrosstalkSources_995848_48.stderr
parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: <AppFuture at 0x14991a457650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 1aab462463e86d630a2b1fe82d03c155
parsl.dataflow.memoization INFO: Task 285 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/20475285-07c4-4176-82fb-9603bab2ea55_isrForCrosstalkSources_995848_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/20475285-07c4-4176-82fb-9603bab2ea55_isrForCrosstalkSources_995848_7.stderr
parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: <AppFuture at 0x14991a457590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 286 has memoization hash b8d52ac46c6c6b42462ad77e3c401c46
parsl.dataflow.memoization INFO: Task 286 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f1fcddd5-d5d3-458e-a412-263a37be3160_isrForCrosstalkSources_995995_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f1fcddd5-d5d3-458e-a412-263a37be3160_isrForCrosstalkSources_995995_36.stderr
parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: <AppFuture at 0x14991a45ccd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 287 has memoization hash caaa20046de055c0258fbbaa36ba0695
parsl.dataflow.memoization INFO: Task 287 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/460cfe00-7b2c-40bb-9544-0008d2620026_isrForCrosstalkSources_995850_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/460cfe00-7b2c-40bb-9544-0008d2620026_isrForCrosstalkSources_995850_20.stderr
parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: <AppFuture at 0x14991a45d690 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 5fabd5e0e1395764ea246a693e8397f9
parsl.dataflow.memoization INFO: Task 288 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/c9a9dc66-f816-4405-a9d8-7559d6a0fdb5_isrForCrosstalkSources_995995_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/c9a9dc66-f816-4405-a9d8-7559d6a0fdb5_isrForCrosstalkSources_995995_14.stderr
parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: <AppFuture at 0x14991a45e390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 78b49792412bc57afe5f411fee489445
parsl.dataflow.memoization INFO: Task 289 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a438fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/a6e1556f-295b-411d-a0a7-b1d15e4a3508_isrForCrosstalkSources_995993_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/a6e1556f-295b-411d-a0a7-b1d15e4a3508_isrForCrosstalkSources_995993_41.stderr
parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: <AppFuture at 0x14991a45d450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 290 has memoization hash d793638d2b7513ac6f1c63f9fb9e1ca0
parsl.dataflow.memoization INFO: Task 290 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c7d2f9db-f00c-4852-a1a8-81d4a18e135f_isrForCrosstalkSources_995849_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c7d2f9db-f00c-4852-a1a8-81d4a18e135f_isrForCrosstalkSources_995849_52.stderr
parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: <AppFuture at 0x14991a45c390 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 291 has memoization hash c74bc3768f673d9eb99cecbfd54b72af
parsl.dataflow.memoization INFO: Task 291 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4391c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e5755a9e-8cb3-46b1-9fae-3744b12d7865_isrForCrosstalkSources_995994_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e5755a9e-8cb3-46b1-9fae-3744b12d7865_isrForCrosstalkSources_995994_39.stderr
parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: <AppFuture at 0x14991a45f050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 292 has memoization hash f7189018200a799452744084d85cde05
parsl.dataflow.memoization INFO: Task 292 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c638904c-019c-46eb-92f5-25e12292769e_isrForCrosstalkSources_995848_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c638904c-019c-46eb-92f5-25e12292769e_isrForCrosstalkSources_995848_45.stderr
parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: <AppFuture at 0x14991a45f250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 293 has memoization hash f7cd543dfb2f81eaacef3b45e2afadf3
parsl.dataflow.memoization INFO: Task 293 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4393a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e5637648-22bc-422a-8926-22be81e5fcad_isrForCrosstalkSources_995848_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e5637648-22bc-422a-8926-22be81e5fcad_isrForCrosstalkSources_995848_25.stderr
parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: <AppFuture at 0x14991a26ca10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 294 has memoization hash ed468954f7953862fed50eeba4d7550a
parsl.dataflow.memoization INFO: Task 294 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ca062d6e-c137-422e-83d9-e408828f5c96_isrForCrosstalkSources_995994_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ca062d6e-c137-422e-83d9-e408828f5c96_isrForCrosstalkSources_995994_56.stderr
parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: <AppFuture at 0x14991a26d4d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 530bae6afc916a94d0ab8e3ec56fa448
parsl.dataflow.memoization INFO: Task 295 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/25034c20-6a03-4990-a96e-3db5bfb189b5_isrForCrosstalkSources_995995_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/25034c20-6a03-4990-a96e-3db5bfb189b5_isrForCrosstalkSources_995995_1.stderr
parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: <AppFuture at 0x14991a26de90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 296 has memoization hash fb74b09029513f8b44dae10f10a37858
parsl.dataflow.memoization INFO: Task 296 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4394e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/5147a3f4-fa08-411a-b496-ff4b3045dd62_isrForCrosstalkSources_995995_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/5147a3f4-fa08-411a-b496-ff4b3045dd62_isrForCrosstalkSources_995995_17.stderr
parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: <AppFuture at 0x14991a26db10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 040ff2d25e24f95e106cc2b784d04957
parsl.dataflow.memoization INFO: Task 297 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4cf762b6-4daf-4b0b-9938-49cd3d1795dc_isrForCrosstalkSources_995994_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4cf762b6-4daf-4b0b-9938-49cd3d1795dc_isrForCrosstalkSources_995994_44.stderr
parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: <AppFuture at 0x14991a26e3d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 3ee38e6a322fc79efda97470b71eca91
parsl.dataflow.memoization INFO: Task 298 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4396c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c5d3e236-3933-448f-a59d-b9659c6d31a0_isrForCrosstalkSources_995993_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c5d3e236-3933-448f-a59d-b9659c6d31a0_isrForCrosstalkSources_995993_34.stderr
parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: <AppFuture at 0x14991a26f490 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 1e171d933817b2296b67790be1063595
parsl.dataflow.memoization INFO: Task 299 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/3bbf8147-4af9-43b5-952e-bbdd38a668a2_isrForCrosstalkSources_995849_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/3bbf8147-4af9-43b5-952e-bbdd38a668a2_isrForCrosstalkSources_995849_28.stderr
parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: <AppFuture at 0x14991a26ee90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 5853d4a86de65734cf5d345ce6ae663b
parsl.dataflow.memoization INFO: Task 300 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/7dbb20ba-386f-4587-9e01-8e61bb1d7a07_isrForCrosstalkSources_995849_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/7dbb20ba-386f-4587-9e01-8e61bb1d7a07_isrForCrosstalkSources_995849_27.stderr
parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: <AppFuture at 0x14991a27c590 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 301 has memoization hash a754c52dd6f18d8b0d2f975c00b8d501
parsl.dataflow.memoization INFO: Task 301 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4e949c00-5249-4acd-8cbf-d22295dd77fb_isrForCrosstalkSources_995850_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4e949c00-5249-4acd-8cbf-d22295dd77fb_isrForCrosstalkSources_995850_59.stderr
parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: <AppFuture at 0x14991a27d0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 4e5c3ec21ad4dca90f03a4d59369139c
parsl.dataflow.memoization INFO: Task 302 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4398a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/57ebbd04-d184-4d28-80e3-970e4e2d352f_isrForCrosstalkSources_995850_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/57ebbd04-d184-4d28-80e3-970e4e2d352f_isrForCrosstalkSources_995850_47.stderr
parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: <AppFuture at 0x14991a27e210 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 4f60cde6cd427e33da9a5486da999a6a
parsl.dataflow.memoization INFO: Task 303 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4399e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/3c4d61d0-3df5-49c7-9111-b3165ddf2845_isrForCrosstalkSources_995849_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/3c4d61d0-3df5-49c7-9111-b3165ddf2845_isrForCrosstalkSources_995849_4.stderr
parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: <AppFuture at 0x14991a27d2d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 304 has memoization hash d85d9c63ed1481e448e95541b7d4d0e7
parsl.dataflow.memoization INFO: Task 304 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/41bfc60a-a6fa-4a72-afe4-daf32a9e92f1_isrForCrosstalkSources_995848_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/41bfc60a-a6fa-4a72-afe4-daf32a9e92f1_isrForCrosstalkSources_995848_54.stderr
parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: <AppFuture at 0x14991a27e650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 305 has memoization hash c65fe74e4c7e1590b79a5b8c6b11b5f9
parsl.dataflow.memoization INFO: Task 305 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/62027339-252a-4075-a070-7e474d65dad2_isrForCrosstalkSources_995994_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/62027339-252a-4075-a070-7e474d65dad2_isrForCrosstalkSources_995994_51.stderr
parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: <AppFuture at 0x14991a27fcd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 25fabeff77742178b3157289fae2221f
parsl.dataflow.memoization INFO: Task 306 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3116a6b9-b118-4188-a3cf-831da4fdf247_isrForCrosstalkSources_995995_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3116a6b9-b118-4188-a3cf-831da4fdf247_isrForCrosstalkSources_995995_57.stderr
parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: <AppFuture at 0x14991a27fd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 67f8e392a854d51c27ce90708c1e1ea6
parsl.dataflow.memoization INFO: Task 307 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/94eaac7c-d870-49d5-a97c-66a83f40b7c9_isrForCrosstalkSources_995995_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/94eaac7c-d870-49d5-a97c-66a83f40b7c9_isrForCrosstalkSources_995995_46.stderr
parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: <AppFuture at 0x14991a428410 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 22f28f828cf3ba103527c8d25f3c86a0
parsl.dataflow.memoization INFO: Task 308 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/e57c06de-1c81-4e96-867e-4523aee7a1af_isrForCrosstalkSources_995850_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/e57c06de-1c81-4e96-867e-4523aee7a1af_isrForCrosstalkSources_995850_38.stderr
parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: <AppFuture at 0x14991a281510 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 9b1ac921d9689279a5c61f090ba048dd
parsl.dataflow.memoization INFO: Task 309 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0c6eb885-24b7-43e2-99b3-f675afeb478d_isrForCrosstalkSources_995995_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0c6eb885-24b7-43e2-99b3-f675afeb478d_isrForCrosstalkSources_995995_37.stderr
parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: <AppFuture at 0x14991a281150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 54d96709a504b902dd5c7a94cb879222
parsl.dataflow.memoization INFO: Task 310 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b0249783-8fde-47a5-9f6b-92205ba8b952_isrForCrosstalkSources_995849_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b0249783-8fde-47a5-9f6b-92205ba8b952_isrForCrosstalkSources_995849_7.stderr
parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: <AppFuture at 0x14991a2826d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 891ebbd7de75bfc7d2226a648ca8622d
parsl.dataflow.memoization INFO: Task 311 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/6bc22b26-35cf-44c6-9ae4-f029eb7f006a_isrForCrosstalkSources_995994_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/6bc22b26-35cf-44c6-9ae4-f029eb7f006a_isrForCrosstalkSources_995994_37.stderr
parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: <AppFuture at 0x14991a282710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 832f2b4e02e07cdd59779a982b682563
parsl.dataflow.memoization INFO: Task 312 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9dc48048-4927-4438-a707-44ccb08aa4d3_isrForCrosstalkSources_995993_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9dc48048-4927-4438-a707-44ccb08aa4d3_isrForCrosstalkSources_995993_36.stderr
parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: <AppFuture at 0x14991a282e10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 313 has memoization hash ce07bd99b7d93d0db95650b856248007
parsl.dataflow.memoization INFO: Task 313 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/2613a58b-18f8-49e6-8aa1-d05fbfe30786_isrForCrosstalkSources_995993_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/2613a58b-18f8-49e6-8aa1-d05fbfe30786_isrForCrosstalkSources_995993_45.stderr
parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: <AppFuture at 0x14991a2839d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 95ccdfb105c17dc460501f191c7866a1
parsl.dataflow.memoization INFO: Task 314 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a439f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/02ef740b-02b4-496a-b043-abb7b3a2e37a_isrForCrosstalkSources_995849_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/02ef740b-02b4-496a-b043-abb7b3a2e37a_isrForCrosstalkSources_995849_29.stderr
parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: <AppFuture at 0x14991a2809d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 4f002a5efb5fdccd7ebd9c5018e256a4
parsl.dataflow.memoization INFO: Task 315 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/edc2277e-fcc7-4d47-9895-5f7ddf8a8459_isrForCrosstalkSources_995994_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/edc2277e-fcc7-4d47-9895-5f7ddf8a8459_isrForCrosstalkSources_995994_16.stderr
parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: <AppFuture at 0x14991a4110d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 10061cc1992ad104628a6bd12b4d5f51
parsl.dataflow.memoization INFO: Task 316 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/dcd48855-eb7e-4d49-ba3a-1306e18b1c4c_isrForCrosstalkSources_995849_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/dcd48855-eb7e-4d49-ba3a-1306e18b1c4c_isrForCrosstalkSources_995849_26.stderr
parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: <AppFuture at 0x14991a288fd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 5868a0079cb607dfc37a22912e647233
parsl.dataflow.memoization INFO: Task 317 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e978b32e-0464-4e63-ba09-b92a7847b38b_isrForCrosstalkSources_995993_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e978b32e-0464-4e63-ba09-b92a7847b38b_isrForCrosstalkSources_995993_57.stderr
parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: <AppFuture at 0x14991a288610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 318 has memoization hash d3a202ab9c639d1259cf553a96e86139
parsl.dataflow.memoization INFO: Task 318 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fd870f20-1620-4f0a-ac0a-6a48081d7e51_isrForCrosstalkSources_995995_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fd870f20-1620-4f0a-ac0a-6a48081d7e51_isrForCrosstalkSources_995995_43.stderr
parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: <AppFuture at 0x14991a289dd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 8607f9e6c8ef17644f0e264bca62beed
parsl.dataflow.memoization INFO: Task 319 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/29deb951-1297-4da6-b59b-ff05958379ac_isrForCrosstalkSources_995995_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/29deb951-1297-4da6-b59b-ff05958379ac_isrForCrosstalkSources_995995_32.stderr
parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: <AppFuture at 0x14991a28b450 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 648ef8e0e80bb222c228a3df6bc14f10
parsl.dataflow.memoization INFO: Task 320 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/e43e1272-f8f3-4d1a-a32f-73c0a50e75e6_isrForCrosstalkSources_995850_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/e43e1272-f8f3-4d1a-a32f-73c0a50e75e6_isrForCrosstalkSources_995850_11.stderr
parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: <AppFuture at 0x14991a28b0d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 6aa759d015d1f16f1695dbd2c2c572a1
parsl.dataflow.memoization INFO: Task 321 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/d42ecdc1-4dc4-48b7-991e-73594ee3e90e_isrForCrosstalkSources_995994_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/d42ecdc1-4dc4-48b7-991e-73594ee3e90e_isrForCrosstalkSources_995994_5.stderr
parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: <AppFuture at 0x14991a28bd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 3c113019fbb67bf9d9441c4e5e13c2a7
parsl.dataflow.memoization INFO: Task 322 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d6e53171-6a06-43b8-bdd4-11c008648106_isrForCrosstalkSources_995850_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d6e53171-6a06-43b8-bdd4-11c008648106_isrForCrosstalkSources_995850_56.stderr
parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: <AppFuture at 0x14991a28b7d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 0943c9ed4453977060a1d3e4135da330
parsl.dataflow.memoization INFO: Task 323 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/c89b2711-279f-4622-bd35-547c7357cdbf_isrForCrosstalkSources_995994_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/c89b2711-279f-4622-bd35-547c7357cdbf_isrForCrosstalkSources_995994_3.stderr
parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: <AppFuture at 0x14991a27c910 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 32c01df6d96b56ee66f65b62c98ec21f
parsl.dataflow.memoization INFO: Task 324 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f27755ef-5817-41c3-a037-86ab1b6a70b0_isrForCrosstalkSources_995995_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f27755ef-5817-41c3-a037-86ab1b6a70b0_isrForCrosstalkSources_995995_24.stderr
parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: <AppFuture at 0x14992617d890 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 48cc62c4538d05ae53de94c9d356e014
parsl.dataflow.memoization INFO: Task 325 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/7e2a64f3-ab88-4fb8-977d-7b232c0cc127_isrForCrosstalkSources_995995_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/7e2a64f3-ab88-4fb8-977d-7b232c0cc127_isrForCrosstalkSources_995995_55.stderr
parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: <AppFuture at 0x14991a294f50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 326 has memoization hash b46e97d1e7d29970604ccb398b63b112
parsl.dataflow.memoization INFO: Task 326 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/cec3d32d-6db6-4219-be57-54eeae93de11_isrForCrosstalkSources_995993_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/cec3d32d-6db6-4219-be57-54eeae93de11_isrForCrosstalkSources_995993_51.stderr
parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: <AppFuture at 0x14991a295ad0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 875817aafc8217778da607b086a2cb26
parsl.dataflow.memoization INFO: Task 327 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/72ad8e77-c5c2-4930-a249-b061512d8d73_isrForCrosstalkSources_995850_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/72ad8e77-c5c2-4930-a249-b061512d8d73_isrForCrosstalkSources_995850_48.stderr
parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: <AppFuture at 0x14991a296bd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 328 has memoization hash aa85dc1559556c2add5dcd9bd710a21e
parsl.dataflow.memoization INFO: Task 328 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/df7e57c6-444e-4a06-b083-189a7fccc6ab_isrForCrosstalkSources_995995_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/df7e57c6-444e-4a06-b083-189a7fccc6ab_isrForCrosstalkSources_995995_44.stderr
parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: <AppFuture at 0x14991a297250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 329 has memoization hash f583d7fafcc4d35cab81a4167e05b30f
parsl.dataflow.memoization INFO: Task 329 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5addadff-b512-4efb-941c-9191bb24b739_isrForCrosstalkSources_995848_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5addadff-b512-4efb-941c-9191bb24b739_isrForCrosstalkSources_995848_26.stderr
parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: <AppFuture at 0x14991a297cd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 4976480e1440c2dd5a7b0a431928a094
parsl.dataflow.memoization INFO: Task 330 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d676e161-eb7e-42dc-a592-068d05a85013_isrForCrosstalkSources_995850_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d676e161-eb7e-42dc-a592-068d05a85013_isrForCrosstalkSources_995850_41.stderr
parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: <AppFuture at 0x14991a297d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 63636ca60e2669f878e1adf0a35431cc
parsl.dataflow.memoization INFO: Task 331 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/52412165-f720-425a-b844-efaf60221448_isrForCrosstalkSources_995849_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/52412165-f720-425a-b844-efaf60221448_isrForCrosstalkSources_995849_56.stderr
parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: <AppFuture at 0x14991a2a8b50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 332 has memoization hash a6c38baa7be1182af6688e4fa2f20956
parsl.dataflow.memoization INFO: Task 332 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/534df77b-f648-4f23-a6a3-351970010611_isrForCrosstalkSources_995993_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/534df77b-f648-4f23-a6a3-351970010611_isrForCrosstalkSources_995993_46.stderr
parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: <AppFuture at 0x14991a2a9d90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 1139b61db5a1cf62d3b25b8d9611f444
parsl.dataflow.memoization INFO: Task 333 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0a212b10-08cb-47f5-91e5-6d3d180b379d_isrForCrosstalkSources_995848_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0a212b10-08cb-47f5-91e5-6d3d180b379d_isrForCrosstalkSources_995848_59.stderr
parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: <AppFuture at 0x14991a2a9150 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 8317fbbf46eae0562cca9f0a3ac666a8
parsl.dataflow.memoization INFO: Task 334 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/708e2038-9c69-49d6-abd1-5389a21c6881_isrForCrosstalkSources_995849_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/708e2038-9c69-49d6-abd1-5389a21c6881_isrForCrosstalkSources_995849_2.stderr
parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: <AppFuture at 0x14991a2aa990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 335 has memoization hash eb70101b2f630dc405ae551667182d7f
parsl.dataflow.memoization INFO: Task 335 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/163c3f9a-9e83-4b7a-8289-173611d2737b_isrForCrosstalkSources_995995_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/163c3f9a-9e83-4b7a-8289-173611d2737b_isrForCrosstalkSources_995995_31.stderr
parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: <AppFuture at 0x14991a2ab810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 336 has memoization hash dd502c5899193c4acd54794648d19d0b
parsl.dataflow.memoization INFO: Task 336 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/2c69f874-d374-44e5-8b64-8725b6fc2022_isrForCrosstalkSources_995993_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/2c69f874-d374-44e5-8b64-8725b6fc2022_isrForCrosstalkSources_995993_19.stderr
parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: <AppFuture at 0x14991a2aafd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 116ae818290f83f0d00ea17263247f0d
parsl.dataflow.memoization INFO: Task 337 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a926d484-ddab-4a6a-a4d1-b9d4a10993de_isrForCrosstalkSources_995995_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a926d484-ddab-4a6a-a4d1-b9d4a10993de_isrForCrosstalkSources_995995_42.stderr
parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: <AppFuture at 0x14991a2aabd0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 338 has memoization hash a99404eeaa69e4078e1ac41f5711a6e7
parsl.dataflow.memoization INFO: Task 338 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/17c65322-78ef-441d-b5b4-67687afdde72_isrForCrosstalkSources_995995_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/17c65322-78ef-441d-b5b4-67687afdde72_isrForCrosstalkSources_995995_39.stderr
parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: <AppFuture at 0x14991a2b0a90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 339 has memoization hash eb4458e1ac572a5364c0779eb2383e35
parsl.dataflow.memoization INFO: Task 339 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/5798c9bb-af40-443d-a399-d3771f7e9576_isrForCrosstalkSources_995849_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/5798c9bb-af40-443d-a399-d3771f7e9576_isrForCrosstalkSources_995849_54.stderr
parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: <AppFuture at 0x14991a2b12d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 4f961f327d56afb8dd8b1202b84562fc
parsl.dataflow.memoization INFO: Task 340 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ef4c03e1-53cf-4d3d-afb3-985dc8b44551_isrForCrosstalkSources_995849_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ef4c03e1-53cf-4d3d-afb3-985dc8b44551_isrForCrosstalkSources_995849_46.stderr
parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: <AppFuture at 0x14991a2b1990 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 341 has memoization hash acb8ffe06ba694d6f815c8e0efa3338f
parsl.dataflow.memoization INFO: Task 341 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/0561368a-2e23-42c6-b65c-43a188fda943_isrForCrosstalkSources_995993_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/0561368a-2e23-42c6-b65c-43a188fda943_isrForCrosstalkSources_995993_31.stderr
parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: <AppFuture at 0x14991a2b2710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 58599a15e7ce7eeaf3a7b1c9f960be48
parsl.dataflow.memoization INFO: Task 342 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/afa5be02-ba30-4f50-8dc1-3cfd774fdccc_isrForCrosstalkSources_995850_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/afa5be02-ba30-4f50-8dc1-3cfd774fdccc_isrForCrosstalkSources_995850_25.stderr
parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: <AppFuture at 0x14991a2b2f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 343 has memoization hash c32318e46a06b6907c999ae1b133adb6
parsl.dataflow.memoization INFO: Task 343 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/76e1b35c-3c83-4ae6-b6e4-05abc468082a_isrForCrosstalkSources_995849_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/76e1b35c-3c83-4ae6-b6e4-05abc468082a_isrForCrosstalkSources_995849_15.stderr
parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: <AppFuture at 0x14991a2b2e90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 00c28e53725bceb1eb3c9cb8b5e55a83
parsl.dataflow.memoization INFO: Task 344 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/b261c2c6-9d28-4051-a8c3-65132590acc7_isrForCrosstalkSources_995848_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/b261c2c6-9d28-4051-a8c3-65132590acc7_isrForCrosstalkSources_995848_58.stderr
parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: <AppFuture at 0x14991a2b3050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 9628c80ffebe96017ebac93d28c77748
parsl.dataflow.memoization INFO: Task 345 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/17d54452-91bf-47a4-83be-138176d76257_isrForCrosstalkSources_995848_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/17d54452-91bf-47a4-83be-138176d76257_isrForCrosstalkSources_995848_50.stderr
parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: <AppFuture at 0x14991a2b3f90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 19b98ddd1de00fe44686ccdf4db17097
parsl.dataflow.memoization INFO: Task 346 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/eab0ebd9-1872-4cec-8925-a305a4bf5fe7_isrForCrosstalkSources_995995_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/eab0ebd9-1872-4cec-8925-a305a4bf5fe7_isrForCrosstalkSources_995995_50.stderr
parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: <AppFuture at 0x14991a2b4810 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 347 has memoization hash dcfea24c52fd2b71e78fd323d512f114
parsl.dataflow.memoization INFO: Task 347 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/7a2027af-7c21-45da-bc96-cfbc64d35092_isrForCrosstalkSources_995995_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/7a2027af-7c21-45da-bc96-cfbc64d35092_isrForCrosstalkSources_995995_60.stderr
parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: <AppFuture at 0x14991a2b58d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 337c4d3b4ea1246793c366b79d4943cb
parsl.dataflow.memoization INFO: Task 348 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/929beb7e-94c9-4357-8e46-e6a0bbc9ccab_isrForCrosstalkSources_995849_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/929beb7e-94c9-4357-8e46-e6a0bbc9ccab_isrForCrosstalkSources_995849_13.stderr
parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: <AppFuture at 0x14991a2b6a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 5642387bc008969486decb20db9fec8b
parsl.dataflow.memoization INFO: Task 349 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2c5045f3-b4ab-4fa7-97e0-82538b05ea58_isrForCrosstalkSources_995995_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2c5045f3-b4ab-4fa7-97e0-82538b05ea58_isrForCrosstalkSources_995995_61.stderr
parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: <AppFuture at 0x14991a2b5d50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 7fba3d05e9ae65c1105c8b55aeec6a61
parsl.dataflow.memoization INFO: Task 350 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/7aeb7c8a-7a87-4900-b492-b24a94791f7e_isrForCrosstalkSources_995848_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/7aeb7c8a-7a87-4900-b492-b24a94791f7e_isrForCrosstalkSources_995848_56.stderr
parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: <AppFuture at 0x14991a2b7250 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 262a77dcf016c8cb40796a0c5732318e
parsl.dataflow.memoization INFO: Task 351 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/bdfe1e4b-73f3-44c8-b55c-7c02a080592f_isrForCrosstalkSources_995993_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/bdfe1e4b-73f3-44c8-b55c-7c02a080592f_isrForCrosstalkSources_995993_49.stderr
parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: <AppFuture at 0x14991a2b78d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 5b7096f1f0955c93f4352137e3e37f8f
parsl.dataflow.memoization INFO: Task 352 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f5959628-b34c-4051-9b88-b8a7007594dd_isrForCrosstalkSources_995995_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f5959628-b34c-4051-9b88-b8a7007594dd_isrForCrosstalkSources_995995_13.stderr
parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: <AppFuture at 0x14991a2b7550 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 0d0640307096e103a7ce40def50f16a8
parsl.dataflow.memoization INFO: Task 353 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bf311e54-3cdd-418e-9444-f7b1ca6a3b04_isrForCrosstalkSources_995850_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bf311e54-3cdd-418e-9444-f7b1ca6a3b04_isrForCrosstalkSources_995850_19.stderr
parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: <AppFuture at 0x14991a2c0a50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 0ffd28f7d45ab26d1df954532adb671b
parsl.dataflow.memoization INFO: Task 354 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/52030bea-9c7c-4cf5-8755-dd8031b39fe8_isrForCrosstalkSources_995850_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/52030bea-9c7c-4cf5-8755-dd8031b39fe8_isrForCrosstalkSources_995850_51.stderr
parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: <AppFuture at 0x14991a2c16d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 5675f3508eb7063150d561355ad39528
parsl.dataflow.memoization INFO: Task 355 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2cdab690-56d5-45f7-ae34-eeec1b6c27e4_isrForCrosstalkSources_995995_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2cdab690-56d5-45f7-ae34-eeec1b6c27e4_isrForCrosstalkSources_995995_12.stderr
parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: <AppFuture at 0x14991a2c2710 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 7b14e8dd1c940575c95da7070eeab1eb
parsl.dataflow.memoization INFO: Task 356 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/45e78ff1-4699-498c-91b5-1a006e8ff33a_isrForCrosstalkSources_995849_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/45e78ff1-4699-498c-91b5-1a006e8ff33a_isrForCrosstalkSources_995849_55.stderr
parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: <AppFuture at 0x14991a2c29d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 1558165efcb9512377f6e7f524de9259
parsl.dataflow.memoization INFO: Task 357 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/74a0b775-6702-477b-8ad5-5f1512acfbba_isrForCrosstalkSources_995994_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/74a0b775-6702-477b-8ad5-5f1512acfbba_isrForCrosstalkSources_995994_47.stderr
parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: <AppFuture at 0x14991a2c2e50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 36b34b88aed76b6cfef9ccef39c2e417
parsl.dataflow.memoization INFO: Task 358 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f783b177-d75a-4532-a35b-52c09a87dde7_isrForCrosstalkSources_995994_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f783b177-d75a-4532-a35b-52c09a87dde7_isrForCrosstalkSources_995994_4.stderr
parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: <AppFuture at 0x14991a2c35d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 90eb9641c8a33b61d146bc0073519af2
parsl.dataflow.memoization INFO: Task 359 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5e7093c3-7995-447b-bc1e-c3f4726413ca_isrForCrosstalkSources_995994_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5e7093c3-7995-447b-bc1e-c3f4726413ca_isrForCrosstalkSources_995994_28.stderr
parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: <AppFuture at 0x14991a2c3c50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 360 has memoization hash b5e6c7fd9148e772126374395fb36294
parsl.dataflow.memoization INFO: Task 360 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d6d73757-28c8-4d09-ac3d-23ec005f1748_isrForCrosstalkSources_995850_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d6d73757-28c8-4d09-ac3d-23ec005f1748_isrForCrosstalkSources_995850_1.stderr
parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: <AppFuture at 0x14991a2cca10 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 4a23e139ce87e645522dce3d20bbadb2
parsl.dataflow.memoization INFO: Task 361 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ba66456c-d6c5-469c-a6af-1c68020afab9_isrForCrosstalkSources_995848_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ba66456c-d6c5-469c-a6af-1c68020afab9_isrForCrosstalkSources_995848_55.stderr
parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: <AppFuture at 0x14991a2cd050 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 692ddacd912e8414c78aa3241fd734f6
parsl.dataflow.memoization INFO: Task 362 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/16fe9034-5770-4aed-8c9f-cee3d987705e_isrForCrosstalkSources_995993_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/16fe9034-5770-4aed-8c9f-cee3d987705e_isrForCrosstalkSources_995993_10.stderr
parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: <AppFuture at 0x14991a2cd750 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 55537c2700488cc2ec89424ebd7baa04
parsl.dataflow.memoization INFO: Task 363 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/eb5d3ba5-aa80-49e2-ad28-c45dfba37884_isrForCrosstalkSources_995994_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/eb5d3ba5-aa80-49e2-ad28-c45dfba37884_isrForCrosstalkSources_995994_62.stderr
parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: <AppFuture at 0x14991a2cdc90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 341ced2364eb47c3a7280221c1533ded
parsl.dataflow.memoization INFO: Task 364 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a43bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/d3014836-44af-462b-ab5f-24b653839644_isrForCrosstalkSources_995993_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/d3014836-44af-462b-ab5f-24b653839644_isrForCrosstalkSources_995993_20.stderr
parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: <AppFuture at 0x14991a2ce650 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 771f3fd9628284c37a09ce57e057e570
parsl.dataflow.memoization INFO: Task 365 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a4380e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/19630532-dd96-4a57-bcba-702087780d18_isrForCrosstalkSources_995850_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/19630532-dd96-4a57-bcba-702087780d18_isrForCrosstalkSources_995850_52.stderr
parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: <AppFuture at 0x14991a2cfd90 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 136d802c7a4904f54a3c403a70ed7d9a
parsl.dataflow.memoization INFO: Task 366 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a2d00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a77487d0-264c-4673-b941-d3e92b38cc21_isrForCrosstalkSources_995994_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a77487d0-264c-4673-b941-d3e92b38cc21_isrForCrosstalkSources_995994_57.stderr
parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: <AppFuture at 0x14991a2cfa50 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 367 has memoization hash b5d8b6b75f6194c498e28f24981012e7
parsl.dataflow.memoization INFO: Task 367 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a2d0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/83c14bfd-3dae-42df-93b5-358a7bf6b0cd_isrForCrosstalkSources_995993_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/83c14bfd-3dae-42df-93b5-358a7bf6b0cd_isrForCrosstalkSources_995993_47.stderr
parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: <AppFuture at 0x14991a2cf610 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 368 has memoization hash da5fd4bd7c04c39f9578312aabfe5065
parsl.dataflow.memoization INFO: Task 368 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a2d02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6452b5cc-9110-40d0-b0b9-a4ffd3a3694c_isrForCrosstalkSources_995993_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6452b5cc-9110-40d0-b0b9-a4ffd3a3694c_isrForCrosstalkSources_995993_44.stderr
parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: <AppFuture at 0x14991a2943d0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 05fc0694500e3d19ec571b521d9c908b
parsl.dataflow.memoization INFO: Task 369 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a2d0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/1fbc3e81-4743-4383-95c5-673d4e8ddbc8_isrForCrosstalkSources_995848_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/1fbc3e81-4743-4383-95c5-673d4e8ddbc8_isrForCrosstalkSources_995848_31.stderr
parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: <AppFuture at 0x14991a2d5ed0 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 370 has memoization hash e307b4487254597a110595233e95cbb9
parsl.dataflow.memoization INFO: Task 370 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a2d0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/47bc7457-77ba-4840-a223-59bd958245d5_isrForCrosstalkSources_995993_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/47bc7457-77ba-4840-a223-59bd958245d5_isrForCrosstalkSources_995993_29.stderr
parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi
parsl.dataflow.dflow DEBUG: Adding output dependencies
parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency
parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: <AppFuture at 0x14991a2d4190 state=pending>
parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout']
parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr
parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout
parsl.dataflow.memoization DEBUG: Task 371 has memoization hash ba92faf33bc0622d6ae4d92c55616c8a
parsl.dataflow.memoization INFO: Task 371 had no result in cache
parsl.executors.high_throughput.executor DEBUG: Pushing function <function wrap_error.<locals>.wrapper at 0x14991a2d0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",)
parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c105e832-2cbe-4bb3-8c20-8df2a3e32231_isrForCrosstalkSources_995993_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c105e832-2cbe-4bb3-8c20-8df2a3e32231_isrForCrosstalkSources_995993_28.stderr
2024-03-15 10:21:37 proc_lsst.multi:146 [INFO]  found job 6561 in provider local
proc_lsst.multi INFO: found job 6561 in provider local
parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors
parsl.dataflow.strategy DEBUG: Strategizing for executor multi
parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 372
parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/0 running/pending blocks, and 2 connected workers
parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism
parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out
parsl.dataflow.strategy DEBUG: Requesting 92 more blocks
parsl.executors.status_handling INFO: Scaling out by 92 blocks
parsl.executors.status_handling INFO: Allocated block ID 1
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1
2024-03-15 10:21:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-1
2024-03-15 10:21:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-15 10:21:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=1 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:38 proc_lsst.multi:170 [INFO]  [multi] job_id 17064169
proc_lsst.multi INFO: [multi] job_id 17064169
2024-03-15 10:21:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:38 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064169
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064169
parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17064169
parsl.executors.status_handling INFO: Allocated block ID 2
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2
2024-03-15 10:21:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-2
2024-03-15 10:21:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=2 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:38 proc_lsst.multi:170 [INFO]  [multi] job_id 17064170
proc_lsst.multi INFO: [multi] job_id 17064170
2024-03-15 10:21:38 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-15 10:21:38 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:38 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064170
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064170
parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17064170
parsl.executors.status_handling INFO: Allocated block ID 3
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3
2024-03-15 10:21:38 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-3
2024-03-15 10:21:38 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:38 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:38 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-15 10:21:38 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:38 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=3 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17064171
proc_lsst.multi INFO: [multi] job_id 17064171
2024-03-15 10:21:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-15 10:21:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:39 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064171
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064171
parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17064171
parsl.executors.status_handling INFO: Allocated block ID 4
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4
2024-03-15 10:21:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-4
2024-03-15 10:21:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-15 10:21:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=4 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17064172
proc_lsst.multi INFO: [multi] job_id 17064172
2024-03-15 10:21:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-15 10:21:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:39 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064172
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064172
parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17064172
parsl.executors.status_handling INFO: Allocated block ID 5
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5
2024-03-15 10:21:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-5
2024-03-15 10:21:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-15 10:21:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=5 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17064173
proc_lsst.multi INFO: [multi] job_id 17064173
2024-03-15 10:21:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:21:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:39 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064173
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064173
parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17064173
parsl.executors.status_handling INFO: Allocated block ID 6
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6
2024-03-15 10:21:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-6
2024-03-15 10:21:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:21:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=6 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17064174
proc_lsst.multi INFO: [multi] job_id 17064174
2024-03-15 10:21:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-15 10:21:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:40 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064174
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064174
parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17064174
parsl.executors.status_handling INFO: Allocated block ID 7
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7
2024-03-15 10:21:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-7
2024-03-15 10:21:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-15 10:21:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=7 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17064175
proc_lsst.multi INFO: [multi] job_id 17064175
2024-03-15 10:21:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-15 10:21:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:40 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064175
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064175
parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17064175
parsl.executors.status_handling INFO: Allocated block ID 8
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8
2024-03-15 10:21:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-8
2024-03-15 10:21:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-15 10:21:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=8 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:40 proc_lsst.multi:170 [INFO]  [multi] job_id 17064176
proc_lsst.multi INFO: [multi] job_id 17064176
2024-03-15 10:21:40 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-15 10:21:40 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:40 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064176
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064176
parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17064176
parsl.executors.status_handling INFO: Allocated block ID 9
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9
2024-03-15 10:21:40 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-9
2024-03-15 10:21:40 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:40 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:40 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-15 10:21:40 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:40 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=9 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17064177
proc_lsst.multi INFO: [multi] job_id 17064177
2024-03-15 10:21:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-15 10:21:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:41 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064177
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064177
parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17064177
parsl.executors.status_handling INFO: Allocated block ID 10
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10
2024-03-15 10:21:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-10
2024-03-15 10:21:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-15 10:21:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=10 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17064178
proc_lsst.multi INFO: [multi] job_id 17064178
2024-03-15 10:21:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-15 10:21:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:41 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064178
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064178
parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17064178
parsl.executors.status_handling INFO: Allocated block ID 11
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11
2024-03-15 10:21:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-11
2024-03-15 10:21:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-15 10:21:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=11 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17064180
proc_lsst.multi INFO: [multi] job_id 17064180
2024-03-15 10:21:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-15 10:21:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064180
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064180
parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17064180
parsl.executors.status_handling INFO: Allocated block ID 12
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12
2024-03-15 10:21:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-12
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=12 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17064181
proc_lsst.multi INFO: [multi] job_id 17064181
2024-03-15 10:21:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-15 10:21:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064181
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064181
parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17064181
parsl.executors.status_handling INFO: Allocated block ID 13
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13
2024-03-15 10:21:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-13
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=13 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17064182
proc_lsst.multi INFO: [multi] job_id 17064182
2024-03-15 10:21:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-15 10:21:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064182
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064182
parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17064182
parsl.executors.status_handling INFO: Allocated block ID 14
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14
2024-03-15 10:21:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-14
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=14 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17064183
proc_lsst.multi INFO: [multi] job_id 17064183
2024-03-15 10:21:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-15 10:21:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064183
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064183
parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17064183
parsl.executors.status_handling INFO: Allocated block ID 15
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15
2024-03-15 10:21:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-15
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=15 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17064184
proc_lsst.multi INFO: [multi] job_id 17064184
2024-03-15 10:21:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-15 10:21:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064184
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064184
parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17064184
parsl.executors.status_handling INFO: Allocated block ID 16
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16
2024-03-15 10:21:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-16
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=16 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17064185
proc_lsst.multi INFO: [multi] job_id 17064185
2024-03-15 10:21:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-15 10:21:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064185
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064185
parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17064185
parsl.executors.status_handling INFO: Allocated block ID 17
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17
2024-03-15 10:21:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-17
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-15 10:21:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=17 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:43 proc_lsst.multi:170 [INFO]  [multi] job_id 17064186
proc_lsst.multi INFO: [multi] job_id 17064186
2024-03-15 10:21:43 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-15 10:21:43 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:43 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064186
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064186
parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17064186
parsl.executors.status_handling INFO: Allocated block ID 18
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18
2024-03-15 10:21:43 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-18
2024-03-15 10:21:43 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:43 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:43 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-15 10:21:43 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:43 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=18 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17064187
proc_lsst.multi INFO: [multi] job_id 17064187
2024-03-15 10:21:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-15 10:21:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:44 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064187
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064187
parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17064187
parsl.executors.status_handling INFO: Allocated block ID 19
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19
2024-03-15 10:21:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-19
2024-03-15 10:21:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-15 10:21:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=19 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17064188
proc_lsst.multi INFO: [multi] job_id 17064188
2024-03-15 10:21:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-15 10:21:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:44 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064188
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064188
parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17064188
parsl.executors.status_handling INFO: Allocated block ID 20
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20
2024-03-15 10:21:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-20
2024-03-15 10:21:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-15 10:21:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=20 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c01ad9bf7dca305c258023e8779f134e with result from task 1
parsl.dataflow.dflow DEBUG: Task 1 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8a98e235-528f-42ed-83cc-18ae44abd87b_isrForCrosstalkSources_995850_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8a98e235-528f-42ed-83cc-18ae44abd87b_isrForCrosstalkSources_995850_34.stderr
parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93fca73b61f92b257568c855bb77bbcc with result from task 0
parsl.dataflow.dflow DEBUG: Task 0 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a98a02a2-2d9f-4666-b195-3c2d2a12bca2_isrForCrosstalkSources_995994_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a98a02a2-2d9f-4666-b195-3c2d2a12bca2_isrForCrosstalkSources_995994_43.stderr
2024-03-15 10:21:46 proc_lsst.multi:170 [INFO]  [multi] job_id 17064189
proc_lsst.multi INFO: [multi] job_id 17064189
2024-03-15 10:21:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:21:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:46 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064189
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064189
parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17064189
parsl.executors.status_handling INFO: Allocated block ID 21
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21
2024-03-15 10:21:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-21
2024-03-15 10:21:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:21:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=21 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:47 proc_lsst.multi:170 [INFO]  [multi] job_id 17064191
proc_lsst.multi INFO: [multi] job_id 17064191
2024-03-15 10:21:47 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-15 10:21:47 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:47 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064191
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064191
parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17064191
parsl.executors.status_handling INFO: Allocated block ID 22
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22
2024-03-15 10:21:47 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-22
2024-03-15 10:21:47 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:47 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:47 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-15 10:21:47 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:47 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=22 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:49 proc_lsst.multi:170 [INFO]  [multi] job_id 17064192
proc_lsst.multi INFO: [multi] job_id 17064192
2024-03-15 10:21:49 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-15 10:21:49 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:49 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064192
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064192
parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17064192
parsl.executors.status_handling INFO: Allocated block ID 23
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23
2024-03-15 10:21:49 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-23
2024-03-15 10:21:49 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:49 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:49 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-15 10:21:49 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:49 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=23 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17064194
proc_lsst.multi INFO: [multi] job_id 17064194
2024-03-15 10:21:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-15 10:21:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:50 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064194
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064194
parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17064194
parsl.executors.status_handling INFO: Allocated block ID 24
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24
2024-03-15 10:21:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-24
2024-03-15 10:21:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-15 10:21:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=24 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:51 proc_lsst.multi:170 [INFO]  [multi] job_id 17064195
proc_lsst.multi INFO: [multi] job_id 17064195
2024-03-15 10:21:51 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-15 10:21:51 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:51 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064195
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064195
parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17064195
parsl.executors.status_handling INFO: Allocated block ID 25
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25
2024-03-15 10:21:51 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-25
2024-03-15 10:21:51 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:51 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:51 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-15 10:21:51 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:51 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=25 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:52 proc_lsst.multi:170 [INFO]  [multi] job_id 17064196
proc_lsst.multi INFO: [multi] job_id 17064196
2024-03-15 10:21:52 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-15 10:21:52 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:52 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064196
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064196
parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17064196
parsl.executors.status_handling INFO: Allocated block ID 26
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26
2024-03-15 10:21:52 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-26
2024-03-15 10:21:52 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:52 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:52 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-15 10:21:52 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:52 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=26 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:21:53 proc_lsst.multi:170 [INFO]  [multi] job_id 17064197
proc_lsst.multi INFO: [multi] job_id 17064197
2024-03-15 10:21:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-15 10:21:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:53 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064197
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064197
parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17064197
parsl.executors.status_handling INFO: Allocated block ID 27
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27
2024-03-15 10:21:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-27
2024-03-15 10:21:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-15 10:21:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=27 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b6a405c159bb1a99dbae7833c871f2c0 with result from task 2
parsl.dataflow.dflow DEBUG: Task 2 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e3e9ff59-3685-470c-8e76-d85b723e8f6c_isrForCrosstalkSources_995994_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e3e9ff59-3685-470c-8e76-d85b723e8f6c_isrForCrosstalkSources_995994_6.stderr
2024-03-15 10:21:55 proc_lsst.multi:170 [INFO]  [multi] job_id 17064200
proc_lsst.multi INFO: [multi] job_id 17064200
2024-03-15 10:21:55 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-15 10:21:55 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:55 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064200
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064200
parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17064200
parsl.executors.status_handling INFO: Allocated block ID 28
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28
2024-03-15 10:21:55 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-28
2024-03-15 10:21:55 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:55 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:55 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-15 10:21:55 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:55 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=28 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 96678af876b922f54a53c4d0209a3d82 with result from task 3
parsl.dataflow.dflow DEBUG: Task 3 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5483a64b-3542-4e8a-99a9-904074e433a3_isrForCrosstalkSources_995994_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5483a64b-3542-4e8a-99a9-904074e433a3_isrForCrosstalkSources_995994_14.stderr
2024-03-15 10:21:56 proc_lsst.multi:170 [INFO]  [multi] job_id 17064202
proc_lsst.multi INFO: [multi] job_id 17064202
2024-03-15 10:21:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-15 10:21:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:56 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064202
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064202
parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17064202
parsl.executors.status_handling INFO: Allocated block ID 29
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29
2024-03-15 10:21:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-29
2024-03-15 10:21:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-15 10:21:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=29 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ef854f12dd4ea33f3a7d92c5d8624d75 with result from task 13
parsl.dataflow.dflow DEBUG: Task 13 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ba0c83f3-ccae-4a60-bdc5-112f744f152c_isrForCrosstalkSources_995994_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ba0c83f3-ccae-4a60-bdc5-112f744f152c_isrForCrosstalkSources_995994_23.stderr
parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3aa2a4065bfed261883942e66b3185d1 with result from task 7
parsl.dataflow.dflow DEBUG: Task 7 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/88bc5c54-da5f-49d9-a529-491ae4f56153_isrForCrosstalkSources_995994_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/88bc5c54-da5f-49d9-a529-491ae4f56153_isrForCrosstalkSources_995994_45.stderr
parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d4daef194d99022b19d52d651e0f5716 with result from task 12
parsl.dataflow.dflow DEBUG: Task 12 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b7cdf59f-60db-499d-b1a0-656102498b3d_isrForCrosstalkSources_995849_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b7cdf59f-60db-499d-b1a0-656102498b3d_isrForCrosstalkSources_995849_49.stderr
parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5649f3a1d123841140886003543e465f with result from task 10
parsl.dataflow.dflow DEBUG: Task 10 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/d639a1b1-86ab-4ffc-b0ca-6441c5d9b833_isrForCrosstalkSources_995995_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/d639a1b1-86ab-4ffc-b0ca-6441c5d9b833_isrForCrosstalkSources_995995_20.stderr
parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ee8518200a50ba986d79b3d01c72a1e5 with result from task 4
parsl.dataflow.dflow DEBUG: Task 4 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/de3b13ac-6226-4951-a2cd-2209dbbb155a_isrForCrosstalkSources_995849_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/de3b13ac-6226-4951-a2cd-2209dbbb155a_isrForCrosstalkSources_995849_57.stderr
parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a60dad045c5cfa6ebe92799265309ea1 with result from task 9
parsl.dataflow.dflow DEBUG: Task 9 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/0c959044-1328-4b2d-8534-525bdc93ceda_isrForCrosstalkSources_995850_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/0c959044-1328-4b2d-8534-525bdc93ceda_isrForCrosstalkSources_995850_16.stderr
parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32ee35563fa01e24ea0f7837da281ba1 with result from task 5
parsl.dataflow.dflow DEBUG: Task 5 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dd6aed3c-4c1a-4334-8e3a-6324a571de4c_isrForCrosstalkSources_995848_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dd6aed3c-4c1a-4334-8e3a-6324a571de4c_isrForCrosstalkSources_995848_51.stderr
parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5bc74a0b4e346020eb74838936e2c497 with result from task 8
parsl.dataflow.dflow DEBUG: Task 8 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/df8a41b0-1e4c-4c2b-94e0-6e0566fa1eb1_isrForCrosstalkSources_995995_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/df8a41b0-1e4c-4c2b-94e0-6e0566fa1eb1_isrForCrosstalkSources_995995_9.stderr
parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f09d78ebbbc0d628a1121f3b32f5f27d with result from task 6
parsl.dataflow.dflow DEBUG: Task 6 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/4f0b73b7-567d-44dc-8e03-ea224a8ca940_isrForCrosstalkSources_995848_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/4f0b73b7-567d-44dc-8e03-ea224a8ca940_isrForCrosstalkSources_995848_18.stderr
2024-03-15 10:21:58 proc_lsst.multi:170 [INFO]  [multi] job_id 17064203
proc_lsst.multi INFO: [multi] job_id 17064203
2024-03-15 10:21:58 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-15 10:21:58 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:58 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064203
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064203
parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17064203
parsl.executors.status_handling INFO: Allocated block ID 30
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30
2024-03-15 10:21:58 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-30
2024-03-15 10:21:58 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:21:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:21:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:21:58 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:21:58 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-15 10:21:58 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:21:58 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=30 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to astro
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 54cb59b074609738cb5ab99356e45c6f with result from task 11
parsl.dataflow.dflow DEBUG: Task 11 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/49facc7d-4535-4b51-9ebd-9dfcbe3935b4_isrForCrosstalkSources_995993_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/49facc7d-4535-4b51-9ebd-9dfcbe3935b4_isrForCrosstalkSources_995993_18.stderr
2024-03-15 10:22:00 proc_lsst.multi:170 [INFO]  [multi] job_id 17064204
proc_lsst.multi INFO: [multi] job_id 17064204
2024-03-15 10:22:00 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:00 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:00 proc_lsst.multi:178 [INFO]  [multi] provider astro accepted submit and returned 17064204
proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17064204
parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17064204
parsl.executors.status_handling INFO: Allocated block ID 31
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31
2024-03-15 10:22:00 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-31
2024-03-15 10:22:00 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:00 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:00 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:00 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-15 10:22:00 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:00 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=31 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:22:02 proc_lsst.multi:170 [INFO]  [multi] job_id 17064206
proc_lsst.multi INFO: [multi] job_id 17064206
2024-03-15 10:22:02 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:02 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:02 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17064206
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17064206
parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17064206
parsl.executors.status_handling INFO: Allocated block ID 32
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32
2024-03-15 10:22:02 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-32
2024-03-15 10:22:02 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:02 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:02 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:02 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:02 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:02 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=32 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:22:03 proc_lsst.multi:170 [INFO]  [multi] job_id 17064208
proc_lsst.multi INFO: [multi] job_id 17064208
2024-03-15 10:22:03 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-15 10:22:03 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:03 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17064208
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17064208
parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17064208
parsl.executors.status_handling INFO: Allocated block ID 33
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33
2024-03-15 10:22:03 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-33
2024-03-15 10:22:03 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:03 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:03 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-15 10:22:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:03 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=33 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8416a4c04a0e935f6ac257958f7af4f0 with result from task 15
parsl.dataflow.dflow DEBUG: Task 15 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/813603c5-e9e2-46f7-9b4c-705dfbf6588b_isrForCrosstalkSources_995993_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/813603c5-e9e2-46f7-9b4c-705dfbf6588b_isrForCrosstalkSources_995993_33.stderr
parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd3d64cd8eba5217737d5e0f2077a945 with result from task 14
parsl.dataflow.dflow DEBUG: Task 14 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/14ade185-6591-42a0-a6f3-774264fd335d_isrForCrosstalkSources_995848_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/14ade185-6591-42a0-a6f3-774264fd335d_isrForCrosstalkSources_995848_11.stderr
parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 35ebb73a51fe6b8014c37c29dee4e21b with result from task 17
parsl.dataflow.dflow DEBUG: Task 17 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/8025ee51-026c-4c0d-aa3c-c8e9ea5187dd_isrForCrosstalkSources_995994_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/8025ee51-026c-4c0d-aa3c-c8e9ea5187dd_isrForCrosstalkSources_995994_20.stderr
parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7bac38910849186596a6e26261eca03b with result from task 16
parsl.dataflow.dflow DEBUG: Task 16 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/f2a00579-7e85-41a4-a204-e1b1a9cb2309_isrForCrosstalkSources_995850_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/f2a00579-7e85-41a4-a204-e1b1a9cb2309_isrForCrosstalkSources_995850_31.stderr
parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cee14c6b0e95eb6ae867ae626787eb3e with result from task 18
parsl.dataflow.dflow DEBUG: Task 18 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/6f2fe2a6-a6a2-46e5-9685-20c46b79f128_isrForCrosstalkSources_995848_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/6f2fe2a6-a6a2-46e5-9685-20c46b79f128_isrForCrosstalkSources_995848_3.stderr
2024-03-15 10:22:05 proc_lsst.multi:170 [INFO]  [multi] job_id 17064209
proc_lsst.multi INFO: [multi] job_id 17064209
2024-03-15 10:22:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-15 10:22:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:05 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17064209
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17064209
parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17064209
parsl.executors.status_handling INFO: Allocated block ID 34
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34
2024-03-15 10:22:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-34
2024-03-15 10:22:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:05 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-15 10:22:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=34 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4005c83b99db7a66e73d582ddbd1f44f with result from task 24
parsl.dataflow.dflow DEBUG: Task 24 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/99e2ac32-d062-4d87-8c74-5b8260233435_isrForCrosstalkSources_995993_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/99e2ac32-d062-4d87-8c74-5b8260233435_isrForCrosstalkSources_995993_6.stderr
parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a814169dda26907d45d03ba0f6d06c29 with result from task 20
parsl.dataflow.dflow DEBUG: Task 20 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/5c9519b9-f6bd-41a7-84c6-3f326c5e6326_isrForCrosstalkSources_995993_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/5c9519b9-f6bd-41a7-84c6-3f326c5e6326_isrForCrosstalkSources_995993_42.stderr
parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4e5c2861832ba7bf2a6f842954478aea with result from task 23
parsl.dataflow.dflow DEBUG: Task 23 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a3327cf7-22ca-48b5-82e8-1638a3fd5ab8_isrForCrosstalkSources_995995_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a3327cf7-22ca-48b5-82e8-1638a3fd5ab8_isrForCrosstalkSources_995995_49.stderr
parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 042d1dfcb81aeb524dfcb7f274bf8c11 with result from task 21
parsl.dataflow.dflow DEBUG: Task 21 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/44fd7d48-eaec-4068-ab0e-3bddeb051d2e_isrForCrosstalkSources_995994_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/44fd7d48-eaec-4068-ab0e-3bddeb051d2e_isrForCrosstalkSources_995994_42.stderr
parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 595521db2919c95438cccd2c3117f042 with result from task 19
parsl.dataflow.dflow DEBUG: Task 19 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c4c82d3f-507a-4a09-947d-2afb9127eff7_isrForCrosstalkSources_995993_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c4c82d3f-507a-4a09-947d-2afb9127eff7_isrForCrosstalkSources_995993_55.stderr
parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b1b91cd180df7c5fe76d901ce58c4dc with result from task 22
parsl.dataflow.dflow DEBUG: Task 22 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9df5789f-b2df-45d3-a967-f1aaed8dc9c5_isrForCrosstalkSources_995850_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9df5789f-b2df-45d3-a967-f1aaed8dc9c5_isrForCrosstalkSources_995850_61.stderr
2024-03-15 10:22:07 proc_lsst.multi:170 [INFO]  [multi] job_id 17064210
proc_lsst.multi INFO: [multi] job_id 17064210
2024-03-15 10:22:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-15 10:22:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:07 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17064210
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17064210
parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17064210
parsl.executors.status_handling INFO: Allocated block ID 35
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35
2024-03-15 10:22:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-35
2024-03-15 10:22:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-15 10:22:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=35 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_rtx6k
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 804cbe7d6fa13f3a55c42276c0d120a8 with result from task 27
parsl.dataflow.dflow DEBUG: Task 27 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6d46ff39-c0cd-4442-9ac7-cfdbbc24767d_isrForCrosstalkSources_995993_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6d46ff39-c0cd-4442-9ac7-cfdbbc24767d_isrForCrosstalkSources_995993_21.stderr
parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 683a7e3e7a9a4920c266e17ca692aff8 with result from task 30
parsl.dataflow.dflow DEBUG: Task 30 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/17659de3-1e1e-4617-aa45-be5a046aeff0_isrForCrosstalkSources_995848_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/17659de3-1e1e-4617-aa45-be5a046aeff0_isrForCrosstalkSources_995848_14.stderr
parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f0e4b987d34ef1c4775221f3dba17b44 with result from task 29
parsl.dataflow.dflow DEBUG: Task 29 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bc704b00-aabe-4263-9c45-d17765988565_isrForCrosstalkSources_995850_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bc704b00-aabe-4263-9c45-d17765988565_isrForCrosstalkSources_995850_12.stderr
parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 72219777fa8c25a4a638f187ad3a638e with result from task 31
parsl.dataflow.dflow DEBUG: Task 31 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a1df886a-001d-4451-bb35-4c0b9e1b7605_isrForCrosstalkSources_995994_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a1df886a-001d-4451-bb35-4c0b9e1b7605_isrForCrosstalkSources_995994_46.stderr
parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89977215285a5790a2a37e673e3d6c34 with result from task 33
parsl.dataflow.dflow DEBUG: Task 33 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3a9953e2-0299-402f-9ab1-135ee708ff3d_isrForCrosstalkSources_995995_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3a9953e2-0299-402f-9ab1-135ee708ff3d_isrForCrosstalkSources_995995_22.stderr
parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 21ef0c1e6932ac3944b7ad9db4bd39a0 with result from task 25
parsl.dataflow.dflow DEBUG: Task 25 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2936d3ef-e0ed-4711-9892-39e425915803_isrForCrosstalkSources_995849_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2936d3ef-e0ed-4711-9892-39e425915803_isrForCrosstalkSources_995849_9.stderr
parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 793ac2168a7b85d18e0f579b7f793500 with result from task 34
parsl.dataflow.dflow DEBUG: Task 34 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1587f614-38ad-4b8e-bc14-79650ac6144d_isrForCrosstalkSources_995849_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1587f614-38ad-4b8e-bc14-79650ac6144d_isrForCrosstalkSources_995849_22.stderr
parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e18e3a326bad9cd65316d3ad60ff6cf0 with result from task 32
parsl.dataflow.dflow DEBUG: Task 32 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b42081c7-2284-4d70-97c2-eef683183d11_isrForCrosstalkSources_995849_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b42081c7-2284-4d70-97c2-eef683183d11_isrForCrosstalkSources_995849_41.stderr
2024-03-15 10:22:08 proc_lsst.multi:170 [INFO]  [multi] job_id 17064211
proc_lsst.multi INFO: [multi] job_id 17064211
2024-03-15 10:22:08 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:08 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:08 proc_lsst.multi:178 [INFO]  [multi] provider escience_rtx6k accepted submit and returned 17064211
proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17064211
parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17064211
parsl.executors.status_handling INFO: Allocated block ID 36
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36
2024-03-15 10:22:08 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-36
2024-03-15 10:22:08 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:08 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:08 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:08 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:08 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-15 10:22:08 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:08 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=36 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9d4e37b9ff6ec44c8471717f042eb899 with result from task 26
parsl.dataflow.dflow DEBUG: Task 26 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/43015391-5126-480b-becd-40bf36bf2038_isrForCrosstalkSources_995994_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/43015391-5126-480b-becd-40bf36bf2038_isrForCrosstalkSources_995994_49.stderr
parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9fce1f3960dc8d67ac02a49fb497ce76 with result from task 28
parsl.dataflow.dflow DEBUG: Task 28 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/7612adc3-1638-425b-8304-667d5559bdb5_isrForCrosstalkSources_995848_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/7612adc3-1638-425b-8304-667d5559bdb5_isrForCrosstalkSources_995848_1.stderr
parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb6e911e5561cf97e6aa0571ebe5f02d with result from task 35
parsl.dataflow.dflow DEBUG: Task 35 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f4ac559e-06ae-42c5-a500-84e4e53bca63_isrForCrosstalkSources_995994_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f4ac559e-06ae-42c5-a500-84e4e53bca63_isrForCrosstalkSources_995994_55.stderr
2024-03-15 10:22:10 proc_lsst.multi:170 [INFO]  [multi] job_id 17064213
proc_lsst.multi INFO: [multi] job_id 17064213
2024-03-15 10:22:10 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:10 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:10 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064213
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064213
parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17064213
parsl.executors.status_handling INFO: Allocated block ID 37
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37
2024-03-15 10:22:10 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-37
2024-03-15 10:22:10 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:10 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:10 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:10 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:10 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:10 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:10 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=37 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:22:12 proc_lsst.multi:170 [INFO]  [multi] job_id 17064214
proc_lsst.multi INFO: [multi] job_id 17064214
2024-03-15 10:22:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-15 10:22:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:12 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064214
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064214
parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17064214
parsl.executors.status_handling INFO: Allocated block ID 38
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38
2024-03-15 10:22:12 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-38
2024-03-15 10:22:12 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:12 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:12 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:12 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-15 10:22:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:12 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=38 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c96197ade847efd7555c1c105ddb3564 with result from task 36
parsl.dataflow.dflow DEBUG: Task 36 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fdb71650-8ddb-479b-a0cb-7537e68073bd_isrForCrosstalkSources_995995_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fdb71650-8ddb-479b-a0cb-7537e68073bd_isrForCrosstalkSources_995995_41.stderr
parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a5e2059e39956ec5e5756d65fa6c9156 with result from task 37
parsl.dataflow.dflow DEBUG: Task 37 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a9b00a3d-63fc-466b-b5c8-6668eea5c551_isrForCrosstalkSources_995994_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a9b00a3d-63fc-466b-b5c8-6668eea5c551_isrForCrosstalkSources_995994_33.stderr
parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 995e1263c617d9c2eed2190129c082b5 with result from task 38
parsl.dataflow.dflow DEBUG: Task 38 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/4534167a-a22d-4328-95fd-b0968dadb786_isrForCrosstalkSources_995849_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/4534167a-a22d-4328-95fd-b0968dadb786_isrForCrosstalkSources_995849_34.stderr
2024-03-15 10:22:14 proc_lsst.multi:170 [INFO]  [multi] job_id 17064215
proc_lsst.multi INFO: [multi] job_id 17064215
2024-03-15 10:22:14 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-15 10:22:14 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:14 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064215
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064215
parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17064215
parsl.executors.status_handling INFO: Allocated block ID 39
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39
2024-03-15 10:22:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-39
2024-03-15 10:22:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:14 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:14 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:14 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-15 10:22:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=39 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d37f0b0def8718f73857ba6876226ccf with result from task 40
parsl.dataflow.dflow DEBUG: Task 40 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cbfed5d7-4985-4886-8b35-53c163f56493_isrForCrosstalkSources_995994_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cbfed5d7-4985-4886-8b35-53c163f56493_isrForCrosstalkSources_995994_58.stderr
parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 40332e02ac0cd0114f7f3fcb3486516c with result from task 42
parsl.dataflow.dflow DEBUG: Task 42 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2902b313-45ea-42b8-987e-318fdac5cb36_isrForCrosstalkSources_995995_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2902b313-45ea-42b8-987e-318fdac5cb36_isrForCrosstalkSources_995995_25.stderr
parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 48c9ce2086c524cff7d98711839c6d59 with result from task 41
parsl.dataflow.dflow DEBUG: Task 41 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/9b456bcd-260c-4e96-95d6-3024bc5db428_isrForCrosstalkSources_995849_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/9b456bcd-260c-4e96-95d6-3024bc5db428_isrForCrosstalkSources_995849_32.stderr
parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry acd9ff658b1c67e180c19c8e208c53d7 with result from task 43
parsl.dataflow.dflow DEBUG: Task 43 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/8888caa2-7ec6-4ae7-9f04-3765fac519ab_isrForCrosstalkSources_995993_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/8888caa2-7ec6-4ae7-9f04-3765fac519ab_isrForCrosstalkSources_995993_16.stderr
parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1be8191681922ed4ae390b93ffe18452 with result from task 44
parsl.dataflow.dflow DEBUG: Task 44 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/601e645a-2ac4-4066-9fcf-aeb6bad6ddc3_isrForCrosstalkSources_995850_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/601e645a-2ac4-4066-9fcf-aeb6bad6ddc3_isrForCrosstalkSources_995850_15.stderr
parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d84674fa4bc5915d5a1a89ce6de285aa with result from task 39
parsl.dataflow.dflow DEBUG: Task 39 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/60767b93-576c-47a9-b842-ac5f21a0bb89_isrForCrosstalkSources_995993_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/60767b93-576c-47a9-b842-ac5f21a0bb89_isrForCrosstalkSources_995993_59.stderr
2024-03-15 10:22:16 proc_lsst.multi:170 [INFO]  [multi] job_id 17064217
proc_lsst.multi INFO: [multi] job_id 17064217
2024-03-15 10:22:16 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-15 10:22:16 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:16 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064217
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064217
parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17064217
parsl.executors.status_handling INFO: Allocated block ID 40
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40
2024-03-15 10:22:16 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-40
2024-03-15 10:22:16 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:16 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:16 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:16 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-15 10:22:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:16 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=40 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d74d5d69ed9ec91527a087cea77b95e with result from task 45
parsl.dataflow.dflow DEBUG: Task 45 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/435afef9-405e-45fc-89b1-2dd748001920_isrForCrosstalkSources_995850_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/435afef9-405e-45fc-89b1-2dd748001920_isrForCrosstalkSources_995850_42.stderr
parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58d0973fb87d9ed150e321312f060647 with result from task 48
parsl.dataflow.dflow DEBUG: Task 48 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/dfe9d51f-542a-4476-8275-a3164de8bd31_isrForCrosstalkSources_995994_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/dfe9d51f-542a-4476-8275-a3164de8bd31_isrForCrosstalkSources_995994_61.stderr
parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c301fa6e296451cb87ba6b22abbbf4fd with result from task 47
parsl.dataflow.dflow DEBUG: Task 47 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1070e4b7-fe62-4d73-ba6b-ba089b794579_isrForCrosstalkSources_995849_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1070e4b7-fe62-4d73-ba6b-ba089b794579_isrForCrosstalkSources_995849_47.stderr
parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad62a31e71cdec5aa336fe9b5ffd3fa5 with result from task 50
parsl.dataflow.dflow DEBUG: Task 50 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/520f2a62-fdc9-4128-a5de-c3984a8e5fcc_isrForCrosstalkSources_995994_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/520f2a62-fdc9-4128-a5de-c3984a8e5fcc_isrForCrosstalkSources_995994_31.stderr
parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7c2d40167ff8696f8faa643e8bc7b99a with result from task 51
parsl.dataflow.dflow DEBUG: Task 51 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/2525ca40-0033-4ab0-8c48-f0be11078b9d_isrForCrosstalkSources_995994_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/2525ca40-0033-4ab0-8c48-f0be11078b9d_isrForCrosstalkSources_995994_19.stderr
parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry adef4a7aedff9e2cdad9af3c2095acd9 with result from task 54
parsl.dataflow.dflow DEBUG: Task 54 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/b8a347b1-e6e0-4fbe-9d7a-7934aa4e4327_isrForCrosstalkSources_995993_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/b8a347b1-e6e0-4fbe-9d7a-7934aa4e4327_isrForCrosstalkSources_995993_24.stderr
parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 34e20fe4bb06ad170ca35472ea5ff7fd with result from task 46
parsl.dataflow.dflow DEBUG: Task 46 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e07ab3a9-94cc-4fae-95be-b65a8d1c0369_isrForCrosstalkSources_995993_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e07ab3a9-94cc-4fae-95be-b65a8d1c0369_isrForCrosstalkSources_995993_30.stderr
parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8b40e110b87f44356be5238ca21b718f with result from task 52
parsl.dataflow.dflow DEBUG: Task 52 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dd15f367-c77b-4c89-bc0e-180b3f2a04ad_isrForCrosstalkSources_995848_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dd15f367-c77b-4c89-bc0e-180b3f2a04ad_isrForCrosstalkSources_995848_36.stderr
parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c8be7809df8bc83638e1e7cd6feea86a with result from task 53
parsl.dataflow.dflow DEBUG: Task 53 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/bda30d32-0e42-4143-bc0e-99df36e4cee7_isrForCrosstalkSources_995848_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/bda30d32-0e42-4143-bc0e-99df36e4cee7_isrForCrosstalkSources_995848_21.stderr
2024-03-15 10:22:18 proc_lsst.multi:170 [INFO]  [multi] job_id 17064218
proc_lsst.multi INFO: [multi] job_id 17064218
2024-03-15 10:22:18 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:18 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:18 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064218
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064218
parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17064218
parsl.executors.status_handling INFO: Allocated block ID 41
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41
2024-03-15 10:22:18 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-41
2024-03-15 10:22:18 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:18 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:18 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:18 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:18 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=41 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89aead8f53f4ceb69fd37d4b0fb397ca with result from task 56
parsl.dataflow.dflow DEBUG: Task 56 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0629d4cb-7bd9-497a-a735-108369eeed7e_isrForCrosstalkSources_995848_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0629d4cb-7bd9-497a-a735-108369eeed7e_isrForCrosstalkSources_995848_41.stderr
parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad770da9d04d00e0ce31e9f9a5919d8d with result from task 49
parsl.dataflow.dflow DEBUG: Task 49 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f42c6e01-cf8d-42a3-ae85-3a5848ba63c8_isrForCrosstalkSources_995994_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f42c6e01-cf8d-42a3-ae85-3a5848ba63c8_isrForCrosstalkSources_995994_24.stderr
parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 18c041828873f05c1ded1676fc1ef49c with result from task 57
parsl.dataflow.dflow DEBUG: Task 57 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5c31adce-642f-438d-9c76-de9ea7da24d4_isrForCrosstalkSources_995848_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5c31adce-642f-438d-9c76-de9ea7da24d4_isrForCrosstalkSources_995848_60.stderr
parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a748d3d27ab01d30445225a2d21873f with result from task 55
parsl.dataflow.dflow DEBUG: Task 55 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fdb58824-29ce-43d8-94fc-7aa8ab1be24f_isrForCrosstalkSources_995995_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fdb58824-29ce-43d8-94fc-7aa8ab1be24f_isrForCrosstalkSources_995995_26.stderr
parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43d1c314cda2ff039563d78e48273838 with result from task 61
parsl.dataflow.dflow DEBUG: Task 61 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/056b47f2-5ef0-41a9-a6b6-3235a07cdc19_isrForCrosstalkSources_995850_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/056b47f2-5ef0-41a9-a6b6-3235a07cdc19_isrForCrosstalkSources_995850_14.stderr
parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bef86267252617049afdfba4bfa7096e with result from task 58
parsl.dataflow.dflow DEBUG: Task 58 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/579078d2-6c75-4fc2-8184-231ae85b1b32_isrForCrosstalkSources_995994_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/579078d2-6c75-4fc2-8184-231ae85b1b32_isrForCrosstalkSources_995994_34.stderr
2024-03-15 10:22:20 proc_lsst.multi:170 [INFO]  [multi] job_id 17064219
proc_lsst.multi INFO: [multi] job_id 17064219
2024-03-15 10:22:20 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-15 10:22:20 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:20 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064219
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064219
parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17064219
parsl.executors.status_handling INFO: Allocated block ID 42
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42
2024-03-15 10:22:20 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-42
2024-03-15 10:22:20 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:20 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:20 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:20 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-15 10:22:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:20 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=42 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0585bb1dc816e7999784cee34a167a1f with result from task 60
parsl.dataflow.dflow DEBUG: Task 60 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/f53bf352-7675-4cfd-a834-595ebd81413c_isrForCrosstalkSources_995993_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/f53bf352-7675-4cfd-a834-595ebd81413c_isrForCrosstalkSources_995993_27.stderr
parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 363dab088c2d4858380c183d698d09ce with result from task 62
parsl.dataflow.dflow DEBUG: Task 62 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/7be2f87b-ac8f-4923-a4bd-3d2b334a102a_isrForCrosstalkSources_995993_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/7be2f87b-ac8f-4923-a4bd-3d2b334a102a_isrForCrosstalkSources_995993_11.stderr
parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8f24425dad89f2993ca829c19cf40f49 with result from task 59
parsl.dataflow.dflow DEBUG: Task 59 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/496735c3-2a63-40a2-8e49-d540dd4a5c38_isrForCrosstalkSources_995848_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/496735c3-2a63-40a2-8e49-d540dd4a5c38_isrForCrosstalkSources_995848_17.stderr
2024-03-15 10:22:22 proc_lsst.multi:170 [INFO]  [multi] job_id 17064220
proc_lsst.multi INFO: [multi] job_id 17064220
2024-03-15 10:22:22 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-15 10:22:22 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:22 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064220
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064220
parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17064220
parsl.executors.status_handling INFO: Allocated block ID 43
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43
2024-03-15 10:22:22 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-43
2024-03-15 10:22:22 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:22 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:22 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:22 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-15 10:22:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:22 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=43 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8fbf93b714b6c96e7c3f9c1697450677 with result from task 63
parsl.dataflow.dflow DEBUG: Task 63 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/8fc311ba-0c20-4eeb-96aa-064c673361eb_isrForCrosstalkSources_995993_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/8fc311ba-0c20-4eeb-96aa-064c673361eb_isrForCrosstalkSources_995993_14.stderr
parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 02843e36656a778dd7ee955b28353555 with result from task 70
parsl.dataflow.dflow DEBUG: Task 70 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cc767d7d-e5d6-49f5-9e76-1e1f3794e1ba_isrForCrosstalkSources_995994_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cc767d7d-e5d6-49f5-9e76-1e1f3794e1ba_isrForCrosstalkSources_995994_29.stderr
2024-03-15 10:22:24 proc_lsst.multi:170 [INFO]  [multi] job_id 17064221
proc_lsst.multi INFO: [multi] job_id 17064221
2024-03-15 10:22:24 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-15 10:22:24 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:24 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064221
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064221
parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17064221
parsl.executors.status_handling INFO: Allocated block ID 44
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44
2024-03-15 10:22:24 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-44
2024-03-15 10:22:24 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:24 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:24 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:24 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:24 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-15 10:22:24 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:24 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=44 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d12dc86e7deae959cfa59afac4986517 with result from task 64
parsl.dataflow.dflow DEBUG: Task 64 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/28a5e8f6-f738-4c6a-8efb-f41d6169b60d_isrForCrosstalkSources_995993_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/28a5e8f6-f738-4c6a-8efb-f41d6169b60d_isrForCrosstalkSources_995993_61.stderr
parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ad90df79d8dd5003efae99f09271c257 with result from task 73
parsl.dataflow.dflow DEBUG: Task 73 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/f90af71d-2482-491c-b334-9b13d4fd007f_isrForCrosstalkSources_995848_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/f90af71d-2482-491c-b334-9b13d4fd007f_isrForCrosstalkSources_995848_46.stderr
parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a6e440374e1c9af7c9dc9347d2e6dcb with result from task 75
parsl.dataflow.dflow DEBUG: Task 75 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a67207ad-fbe3-4a2f-93c9-9719479e0002_isrForCrosstalkSources_995994_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a67207ad-fbe3-4a2f-93c9-9719479e0002_isrForCrosstalkSources_995994_41.stderr
parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f45fcfb82ef4b70f27012ba3160291d7 with result from task 74
parsl.dataflow.dflow DEBUG: Task 74 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/21519b77-8e6e-447a-b516-7d6c89753dc7_isrForCrosstalkSources_995850_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/21519b77-8e6e-447a-b516-7d6c89753dc7_isrForCrosstalkSources_995850_46.stderr
parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 019bb589f7aecb85fe8c2f95d3a716b4 with result from task 72
parsl.dataflow.dflow DEBUG: Task 72 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fc434b24-7c64-4504-a7ef-0e710f4c9938_isrForCrosstalkSources_995995_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fc434b24-7c64-4504-a7ef-0e710f4c9938_isrForCrosstalkSources_995995_30.stderr
2024-03-15 10:22:26 proc_lsst.multi:170 [INFO]  [multi] job_id 17064222
proc_lsst.multi INFO: [multi] job_id 17064222
2024-03-15 10:22:26 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-15 10:22:26 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:26 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064222
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064222
parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17064222
parsl.executors.status_handling INFO: Allocated block ID 45
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45
2024-03-15 10:22:26 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-45
2024-03-15 10:22:26 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:26 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:26 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:26 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-15 10:22:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:26 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=45 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a75a969e9ff788dfa6d7cbd3c11dee5b with result from task 76
parsl.dataflow.dflow DEBUG: Task 76 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/620fdf74-d2e8-4f4f-8311-629d02b67f17_isrForCrosstalkSources_995995_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/620fdf74-d2e8-4f4f-8311-629d02b67f17_isrForCrosstalkSources_995995_35.stderr
parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6851198a38fb9d443cf28bb4ae5de9f2 with result from task 78
parsl.dataflow.dflow DEBUG: Task 78 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c5705c8f-789c-41a8-b6bd-a53393819441_isrForCrosstalkSources_995848_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c5705c8f-789c-41a8-b6bd-a53393819441_isrForCrosstalkSources_995848_38.stderr
parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a2d9a4bb76b01d64c91a16d3d2bd6260 with result from task 77
parsl.dataflow.dflow DEBUG: Task 77 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/bad4e1ee-3932-4f37-8b86-f0a42867afc9_isrForCrosstalkSources_995994_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/bad4e1ee-3932-4f37-8b86-f0a42867afc9_isrForCrosstalkSources_995994_12.stderr
parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b4d331659d72a3018218ac2e81bb984e with result from task 79
parsl.dataflow.dflow DEBUG: Task 79 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/92302f8c-a532-40e9-9467-d2e4a9b0ea47_isrForCrosstalkSources_995993_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/92302f8c-a532-40e9-9467-d2e4a9b0ea47_isrForCrosstalkSources_995993_12.stderr
parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c681e5d2642cab13c4c72cc8150d25e9 with result from task 84
parsl.dataflow.dflow DEBUG: Task 84 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/974621c6-896d-45d9-9864-6ede98b9e2d2_isrForCrosstalkSources_995850_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/974621c6-896d-45d9-9864-6ede98b9e2d2_isrForCrosstalkSources_995850_32.stderr
parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry de88f405776e281f7a0c9734395eccd3 with result from task 81
parsl.dataflow.dflow DEBUG: Task 81 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/710aec35-3bad-4fdc-932e-023c88decabd_isrForCrosstalkSources_995850_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/710aec35-3bad-4fdc-932e-023c88decabd_isrForCrosstalkSources_995850_2.stderr
parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2a436ddb9b4e7660f49b5bf1450a0da8 with result from task 80
parsl.dataflow.dflow DEBUG: Task 80 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bfc63bbc-2858-4b2e-887e-e71a37d78016_isrForCrosstalkSources_995850_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bfc63bbc-2858-4b2e-887e-e71a37d78016_isrForCrosstalkSources_995850_43.stderr
parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 70db110f67e79bb7126ce27f025b2449 with result from task 83
parsl.dataflow.dflow DEBUG: Task 83 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e3fd52c0-d6b5-4787-a6ad-0a778746554e_isrForCrosstalkSources_995848_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e3fd52c0-d6b5-4787-a6ad-0a778746554e_isrForCrosstalkSources_995848_16.stderr
parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 432baa5268c6c609e7c4c6f3b5621511 with result from task 86
parsl.dataflow.dflow DEBUG: Task 86 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e497f482-4e5e-41de-99d6-4b836f06ca9f_isrForCrosstalkSources_995994_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e497f482-4e5e-41de-99d6-4b836f06ca9f_isrForCrosstalkSources_995994_18.stderr
parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8004ceee9e44da5fdf3d70b5b89b670b with result from task 87
parsl.dataflow.dflow DEBUG: Task 87 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/1f7f0304-04ed-4a86-85b8-361a1f42ad9a_isrForCrosstalkSources_995995_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/1f7f0304-04ed-4a86-85b8-361a1f42ad9a_isrForCrosstalkSources_995995_8.stderr
2024-03-15 10:22:28 proc_lsst.multi:170 [INFO]  [multi] job_id 17064223
proc_lsst.multi INFO: [multi] job_id 17064223
2024-03-15 10:22:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-15 10:22:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:28 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064223
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064223
parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17064223
parsl.executors.status_handling INFO: Allocated block ID 46
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46
2024-03-15 10:22:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-46
2024-03-15 10:22:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:28 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:28 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-15 10:22:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=46 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c5e59026211390ef2f58e863f202c652 with result from task 89
parsl.dataflow.dflow DEBUG: Task 89 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0edde73c-2a95-4e0d-873c-451890054978_isrForCrosstalkSources_995995_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0edde73c-2a95-4e0d-873c-451890054978_isrForCrosstalkSources_995995_5.stderr
parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0445a99289f77ec5583f15868ec0fc0f with result from task 91
parsl.dataflow.dflow DEBUG: Task 91 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/313b574c-b1d5-4ec6-872b-33cfdab3cbb3_isrForCrosstalkSources_995994_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/313b574c-b1d5-4ec6-872b-33cfdab3cbb3_isrForCrosstalkSources_995994_15.stderr
parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 326c57cdc5df5582a9109a32078f3676 with result from task 88
parsl.dataflow.dflow DEBUG: Task 88 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2b0a8d19-5776-477f-ad63-f9ce3c602dd9_isrForCrosstalkSources_995995_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2b0a8d19-5776-477f-ad63-f9ce3c602dd9_isrForCrosstalkSources_995995_56.stderr
parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80ebd0bd90d7ffe9ab2032589d2c55c0 with result from task 90
parsl.dataflow.dflow DEBUG: Task 90 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/4beba3cb-e25b-48b4-852f-11854b53c436_isrForCrosstalkSources_995993_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/4beba3cb-e25b-48b4-852f-11854b53c436_isrForCrosstalkSources_995993_48.stderr
parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 758153486b6c5fe9561682da49cbe1af with result from task 93
parsl.dataflow.dflow DEBUG: Task 93 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f87eeefc-75d2-47cf-8d67-05bc23d2433f_isrForCrosstalkSources_995995_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f87eeefc-75d2-47cf-8d67-05bc23d2433f_isrForCrosstalkSources_995995_33.stderr
parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1de8c602c65ed2fb8223054b482ed1c0 with result from task 92
parsl.dataflow.dflow DEBUG: Task 92 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/18389d78-28e5-402e-ba89-efa9df3a6879_isrForCrosstalkSources_995995_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/18389d78-28e5-402e-ba89-efa9df3a6879_isrForCrosstalkSources_995995_15.stderr
parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d8bd992230cfab78d1bae8e2dc2281f2 with result from task 82
parsl.dataflow.dflow DEBUG: Task 82 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1c9ba72e-ea4a-4dbc-a191-50a0979fcbd4_isrForCrosstalkSources_995849_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1c9ba72e-ea4a-4dbc-a191-50a0979fcbd4_isrForCrosstalkSources_995849_14.stderr
parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 92e741e3fbf14e92c1729e63b1610237 with result from task 95
parsl.dataflow.dflow DEBUG: Task 95 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/31070305-650c-465f-a050-f52c0d3846f6_isrForCrosstalkSources_995993_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/31070305-650c-465f-a050-f52c0d3846f6_isrForCrosstalkSources_995993_60.stderr
parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d2e82b6499c96825c37b43e9eb638af6 with result from task 85
parsl.dataflow.dflow DEBUG: Task 85 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/fd909c8a-288e-4a3f-9ab1-0fcec0f40f8f_isrForCrosstalkSources_995850_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/fd909c8a-288e-4a3f-9ab1-0fcec0f40f8f_isrForCrosstalkSources_995850_6.stderr
2024-03-15 10:22:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17064224
proc_lsst.multi INFO: [multi] job_id 17064224
2024-03-15 10:22:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-15 10:22:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:30 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064224
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064224
parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17064224
parsl.executors.status_handling INFO: Allocated block ID 47
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47
2024-03-15 10:22:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-47
2024-03-15 10:22:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:30 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:30 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-15 10:22:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=47 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32d94d2d14da4255bb683c2fe194adc6 with result from task 96
parsl.dataflow.dflow DEBUG: Task 96 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c7fb1623-51d2-4585-af0b-99aad85a25a1_isrForCrosstalkSources_995850_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c7fb1623-51d2-4585-af0b-99aad85a25a1_isrForCrosstalkSources_995850_36.stderr
parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8d7126091a19407dc8f0782036d16705 with result from task 94
parsl.dataflow.dflow DEBUG: Task 94 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/b9e15fcf-05d3-44b8-a689-d554f7e97c08_isrForCrosstalkSources_995850_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/b9e15fcf-05d3-44b8-a689-d554f7e97c08_isrForCrosstalkSources_995850_58.stderr
2024-03-15 10:22:31 proc_lsst.multi:170 [INFO]  [multi] job_id 17064228
proc_lsst.multi INFO: [multi] job_id 17064228
2024-03-15 10:22:31 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-15 10:22:31 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:31 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064228
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064228
parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17064228
parsl.executors.status_handling INFO: Allocated block ID 48
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48
2024-03-15 10:22:31 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-48
2024-03-15 10:22:31 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:31 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:31 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:31 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:31 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-15 10:22:31 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:31 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=48 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 561951e89ccc748a1e959f34699dae21 with result from task 98
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow DEBUG: Task 98 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/276376e4-025f-4d6e-bf4f-bc8b4a203689_isrForCrosstalkSources_995994_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/276376e4-025f-4d6e-bf4f-bc8b4a203689_isrForCrosstalkSources_995994_40.stderr
parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e6379ee0c3aaa86a561951a685ec7236 with result from task 97
parsl.dataflow.dflow DEBUG: Task 97 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/1181f88d-f618-4776-aea6-f679edbf3e48_isrForCrosstalkSources_995993_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/1181f88d-f618-4776-aea6-f679edbf3e48_isrForCrosstalkSources_995993_5.stderr
parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1338e920c8513608ee504fd338c7b455 with result from task 99
parsl.dataflow.dflow DEBUG: Task 99 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/ede051e6-f070-4469-885e-6c7ff26c7e8a_isrForCrosstalkSources_995995_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/ede051e6-f070-4469-885e-6c7ff26c7e8a_isrForCrosstalkSources_995995_52.stderr
parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3bdc31ba8b30bde73190c171894912f0 with result from task 100
parsl.dataflow.dflow DEBUG: Task 100 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/79f4c140-c19e-4b23-a692-d0f95a3bc8d7_isrForCrosstalkSources_995993_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/79f4c140-c19e-4b23-a692-d0f95a3bc8d7_isrForCrosstalkSources_995993_35.stderr
2024-03-15 10:22:33 proc_lsst.multi:170 [INFO]  [multi] job_id 17064233
proc_lsst.multi INFO: [multi] job_id 17064233
2024-03-15 10:22:33 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-15 10:22:33 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:33 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064233
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064233
parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17064233
parsl.executors.status_handling INFO: Allocated block ID 49
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49
2024-03-15 10:22:33 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-49
2024-03-15 10:22:33 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:33 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:33 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:33 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:33 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-15 10:22:33 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:33 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=49 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a196fb5aa8fcf4ec8c3045b496a3f0ac with result from task 101
parsl.dataflow.dflow DEBUG: Task 101 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/cb1d3409-4a14-4aec-a4fe-cd5268da7a8d_isrForCrosstalkSources_995848_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/cb1d3409-4a14-4aec-a4fe-cd5268da7a8d_isrForCrosstalkSources_995848_35.stderr
parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 947b15f80d340b24bc01318732b72b03 with result from task 102
parsl.dataflow.dflow DEBUG: Task 102 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a1237ff5-e482-46d0-aabd-b80e9300337c_isrForCrosstalkSources_995848_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a1237ff5-e482-46d0-aabd-b80e9300337c_isrForCrosstalkSources_995848_9.stderr
parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c7c302fcec884228cd98a03beb05c827 with result from task 69
parsl.dataflow.dflow DEBUG: Task 69 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dcaf2fa3-7807-43ce-93f7-d8e58ef9f2ed_isrForCrosstalkSources_995848_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dcaf2fa3-7807-43ce-93f7-d8e58ef9f2ed_isrForCrosstalkSources_995848_19.stderr
parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6a4e7041a0a1df92b94179c7a590dfc6 with result from task 104
parsl.dataflow.dflow DEBUG: Task 104 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/6b9a7b91-b7b7-42df-9866-6c777094fb17_isrForCrosstalkSources_995848_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/6b9a7b91-b7b7-42df-9866-6c777094fb17_isrForCrosstalkSources_995848_53.stderr
2024-03-15 10:22:35 proc_lsst.multi:170 [INFO]  [multi] job_id 17064234
proc_lsst.multi INFO: [multi] job_id 17064234
2024-03-15 10:22:35 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-15 10:22:35 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:35 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064234
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064234
parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17064234
parsl.executors.status_handling INFO: Allocated block ID 50
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50
2024-03-15 10:22:35 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-50
2024-03-15 10:22:35 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:35 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:35 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:35 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:35 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-15 10:22:35 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:35 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=50 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 998a8972ace790be9c0a4166c8ed0155 with result from task 103
parsl.dataflow.dflow DEBUG: Task 103 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/99e3f1f9-780b-469d-800c-a378a5dd0d18_isrForCrosstalkSources_995994_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/99e3f1f9-780b-469d-800c-a378a5dd0d18_isrForCrosstalkSources_995994_35.stderr
parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 09a1270fb9ba5d47844d90f129f5faeb with result from task 105
parsl.dataflow.dflow DEBUG: Task 105 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1a946e96-7211-46e9-9c24-6c8f5a4c82e1_isrForCrosstalkSources_995849_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1a946e96-7211-46e9-9c24-6c8f5a4c82e1_isrForCrosstalkSources_995849_50.stderr
parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb24b5a271c5ed005c9605d0e12daa1c with result from task 110
parsl.dataflow.dflow DEBUG: Task 110 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/871fd4d0-5cc3-4d4b-96c7-9c8404f72fcc_isrForCrosstalkSources_995994_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/871fd4d0-5cc3-4d4b-96c7-9c8404f72fcc_isrForCrosstalkSources_995994_10.stderr
parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 46cb2297eb73d46e2b4e616957f9b098 with result from task 111
parsl.dataflow.dflow DEBUG: Task 111 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/13b19db7-bf26-4143-8d7b-4fd821cb5829_isrForCrosstalkSources_995993_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/13b19db7-bf26-4143-8d7b-4fd821cb5829_isrForCrosstalkSources_995993_4.stderr
parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 99c7acd8324d811edcab3f01c9bb9783 with result from task 67
parsl.dataflow.dflow DEBUG: Task 67 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/50bb205a-cb68-44c1-9472-c92ef274b372_isrForCrosstalkSources_995993_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/50bb205a-cb68-44c1-9472-c92ef274b372_isrForCrosstalkSources_995993_9.stderr
parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0718e451e8964d146b36836d248cca48 with result from task 112
parsl.dataflow.dflow DEBUG: Task 112 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c111801c-d93e-4f32-9783-82b1f285508e_isrForCrosstalkSources_995850_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c111801c-d93e-4f32-9783-82b1f285508e_isrForCrosstalkSources_995850_60.stderr
parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2b5efd4fc325260b5b6525cc5cbc67e4 with result from task 113
parsl.dataflow.dflow DEBUG: Task 113 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9f070c5e-5a0a-4e47-8d2f-2412b1b1e7f3_isrForCrosstalkSources_995993_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9f070c5e-5a0a-4e47-8d2f-2412b1b1e7f3_isrForCrosstalkSources_995993_56.stderr
parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9de13cb58e11c889cb9fbfc02a398206 with result from task 65
parsl.dataflow.dflow DEBUG: Task 65 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/d95f0956-b08d-4d7c-ba23-f39fba457d62_isrForCrosstalkSources_995993_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/d95f0956-b08d-4d7c-ba23-f39fba457d62_isrForCrosstalkSources_995993_7.stderr
2024-03-15 10:22:37 proc_lsst.multi:170 [INFO]  [multi] job_id 17064239
proc_lsst.multi INFO: [multi] job_id 17064239
2024-03-15 10:22:37 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-15 10:22:37 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:37 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064239
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064239
parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17064239
parsl.executors.status_handling INFO: Allocated block ID 51
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51
2024-03-15 10:22:37 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-51
2024-03-15 10:22:37 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:37 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:37 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:37 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:37 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-15 10:22:37 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:37 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=51 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f424550960e3eccd23c88c81a7342f4 with result from task 71
parsl.dataflow.dflow DEBUG: Task 71 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9879411d-d575-4501-a196-8aee0c902967_isrForCrosstalkSources_995850_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9879411d-d575-4501-a196-8aee0c902967_isrForCrosstalkSources_995850_28.stderr
parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f673ba8c54c5761c0bcbb849c133cf5e with result from task 115
parsl.dataflow.dflow DEBUG: Task 115 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9d15e8a9-62fa-408b-a849-fa486aeba5df_isrForCrosstalkSources_995995_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9d15e8a9-62fa-408b-a849-fa486aeba5df_isrForCrosstalkSources_995995_27.stderr
parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 944a3788fb4f43999f739e351f67dd48 with result from task 114
parsl.dataflow.dflow DEBUG: Task 114 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/b0f43500-ed04-484b-bd6f-8191a5d517cf_isrForCrosstalkSources_995993_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/b0f43500-ed04-484b-bd6f-8191a5d517cf_isrForCrosstalkSources_995993_50.stderr
parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 280bd72c21cc9176ccfd3df042190045 with result from task 120
parsl.dataflow.dflow DEBUG: Task 120 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/368a341a-725a-4031-9c59-6a7ea01418e4_isrForCrosstalkSources_995849_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/368a341a-725a-4031-9c59-6a7ea01418e4_isrForCrosstalkSources_995849_61.stderr
parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd8a0106a5d7fb4f3dea514b70ae3a83 with result from task 68
parsl.dataflow.dflow DEBUG: Task 68 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/4eb269d2-0a46-407b-901b-9e6f178595a3_isrForCrosstalkSources_995993_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/4eb269d2-0a46-407b-901b-9e6f178595a3_isrForCrosstalkSources_995993_22.stderr
parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 624fb369bec5dbec048f8e398c26e7f5 with result from task 119
parsl.dataflow.dflow DEBUG: Task 119 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/ee8ff856-be56-47f5-bd4d-9d133e837291_isrForCrosstalkSources_995850_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/ee8ff856-be56-47f5-bd4d-9d133e837291_isrForCrosstalkSources_995850_10.stderr
parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 70fa3efd2733968975201ca419470461 with result from task 66
parsl.dataflow.dflow DEBUG: Task 66 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/e5d8e6e4-e718-4f6b-b6a3-4a57006fddef_isrForCrosstalkSources_995995_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/e5d8e6e4-e718-4f6b-b6a3-4a57006fddef_isrForCrosstalkSources_995995_40.stderr
parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3d70be0d3cb6373e3cb4f50383bc257c with result from task 116
parsl.dataflow.dflow DEBUG: Task 116 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c7f0d776-e002-4ae8-b756-0a4246c1d2fb_isrForCrosstalkSources_995850_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/c7f0d776-e002-4ae8-b756-0a4246c1d2fb_isrForCrosstalkSources_995850_49.stderr
parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c855e7a7a178949d79085c1c00351adf with result from task 118
parsl.dataflow.dflow DEBUG: Task 118 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/b5a773d5-13d0-48f4-b017-02bd9e7462a2_isrForCrosstalkSources_995994_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/b5a773d5-13d0-48f4-b017-02bd9e7462a2_isrForCrosstalkSources_995994_7.stderr
parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fa27600603f15b5c1c55513185cdbc42 with result from task 121
parsl.dataflow.dflow DEBUG: Task 121 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/129df106-3a00-4475-b4f2-93f4ceb61ed4_isrForCrosstalkSources_995850_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/129df106-3a00-4475-b4f2-93f4ceb61ed4_isrForCrosstalkSources_995850_3.stderr
parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4fb01dc79eec1e2dc7656a1e56044609 with result from task 122
parsl.dataflow.dflow DEBUG: Task 122 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c71c7bc9-d085-4cad-97b8-f625e5f8a4f5_isrForCrosstalkSources_995849_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c71c7bc9-d085-4cad-97b8-f625e5f8a4f5_isrForCrosstalkSources_995849_12.stderr
2024-03-15 10:22:39 proc_lsst.multi:170 [INFO]  [multi] job_id 17064240
proc_lsst.multi INFO: [multi] job_id 17064240
2024-03-15 10:22:39 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-15 10:22:39 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:39 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064240
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064240
parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17064240
parsl.executors.status_handling INFO: Allocated block ID 52
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52
2024-03-15 10:22:39 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-52
2024-03-15 10:22:39 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:39 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:39 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:39 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:39 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-15 10:22:39 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:39 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=52 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 06e1e04407716d8a4b9b3ce6c275c9a9 with result from task 117
parsl.dataflow.dflow DEBUG: Task 117 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/f98cb30a-cc90-431f-bd20-6976ba452936_isrForCrosstalkSources_995849_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/f98cb30a-cc90-431f-bd20-6976ba452936_isrForCrosstalkSources_995849_43.stderr
parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0e1776abe4683c3b081f6a9761ec3dca with result from task 124
parsl.dataflow.dflow DEBUG: Task 124 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3bce7f8c-477f-4be9-84e2-42a930fa6342_isrForCrosstalkSources_995995_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3bce7f8c-477f-4be9-84e2-42a930fa6342_isrForCrosstalkSources_995995_34.stderr
parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 430100961a82d6457c616f2d3f0320f6 with result from task 123
parsl.dataflow.dflow DEBUG: Task 123 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0e78f02c-d1d3-4e93-9b8c-a5630659edf7_isrForCrosstalkSources_995995_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0e78f02c-d1d3-4e93-9b8c-a5630659edf7_isrForCrosstalkSources_995995_51.stderr
parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e933f6e169cd70c0ed8622f78c041ec1 with result from task 125
parsl.dataflow.dflow DEBUG: Task 125 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/10fc11ac-29ee-4e50-ab84-35f8277d7818_isrForCrosstalkSources_995849_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/10fc11ac-29ee-4e50-ab84-35f8277d7818_isrForCrosstalkSources_995849_36.stderr
parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bc971fbde3dc4c882bcdc6f2a1fb7ad5 with result from task 127
parsl.dataflow.dflow DEBUG: Task 127 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/ef658202-d368-4f67-9a09-a394c42f2f30_isrForCrosstalkSources_995993_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/ef658202-d368-4f67-9a09-a394c42f2f30_isrForCrosstalkSources_995993_52.stderr
parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6f8dc4dbef39915170fde28de81fddbb with result from task 129
parsl.dataflow.dflow DEBUG: Task 129 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/77c938e3-21a4-43d3-9ef5-fc058f9d6bf7_isrForCrosstalkSources_995850_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/77c938e3-21a4-43d3-9ef5-fc058f9d6bf7_isrForCrosstalkSources_995850_44.stderr
parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f824dac351af9c04eb7a9eef1b7d926f with result from task 130
parsl.dataflow.dflow DEBUG: Task 130 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/12b05366-7f24-4fc9-a6a4-26af10861463_isrForCrosstalkSources_995850_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/12b05366-7f24-4fc9-a6a4-26af10861463_isrForCrosstalkSources_995850_17.stderr
parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43a19de43c0aa6c858bd84308b9d77e0 with result from task 126
parsl.dataflow.dflow DEBUG: Task 126 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/4639780e-ed06-4a21-bdd8-3e4eafded986_isrForCrosstalkSources_995995_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/4639780e-ed06-4a21-bdd8-3e4eafded986_isrForCrosstalkSources_995995_10.stderr
2024-03-15 10:22:41 proc_lsst.multi:170 [INFO]  [multi] job_id 17064241
proc_lsst.multi INFO: [multi] job_id 17064241
2024-03-15 10:22:41 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-15 10:22:41 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:41 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064241
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064241
parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17064241
parsl.executors.status_handling INFO: Allocated block ID 53
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53
2024-03-15 10:22:41 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-53
2024-03-15 10:22:41 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:41 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:41 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:41 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:41 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-15 10:22:41 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:41 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=53 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0a5340d249f644171c256a22612b0810 with result from task 128
parsl.dataflow.dflow DEBUG: Task 128 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/031bbd43-8ccf-4ca6-8dbc-a8fadfd8e57c_isrForCrosstalkSources_995994_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/031bbd43-8ccf-4ca6-8dbc-a8fadfd8e57c_isrForCrosstalkSources_995994_59.stderr
parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c164db9fc4a6c229e88fe865b9bfaeec with result from task 132
parsl.dataflow.dflow DEBUG: Task 132 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/73ff802b-76c8-4d64-9f7f-78d8de81d789_isrForCrosstalkSources_995993_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/73ff802b-76c8-4d64-9f7f-78d8de81d789_isrForCrosstalkSources_995993_26.stderr
2024-03-15 10:22:42 proc_lsst.multi:170 [INFO]  [multi] job_id 17064242
proc_lsst.multi INFO: [multi] job_id 17064242
2024-03-15 10:22:42 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-15 10:22:42 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:42 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064242
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064242
parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17064242
parsl.executors.status_handling INFO: Allocated block ID 54
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54
2024-03-15 10:22:42 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-54
2024-03-15 10:22:42 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:42 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:42 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:42 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:42 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-15 10:22:42 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:42 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=54 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 88d07c3002651f0666c0e1c3f8fea83e with result from task 133
parsl.dataflow.dflow DEBUG: Task 133 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4e8c55cd-953a-4f83-8f59-63694c2816b3_isrForCrosstalkSources_995994_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4e8c55cd-953a-4f83-8f59-63694c2816b3_isrForCrosstalkSources_995994_2.stderr
parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7159d867575def6f348fd54a8048b5fa with result from task 134
parsl.dataflow.dflow DEBUG: Task 134 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/293dd85d-7d5d-4d58-abb7-0cd460e87819_isrForCrosstalkSources_995995_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/293dd85d-7d5d-4d58-abb7-0cd460e87819_isrForCrosstalkSources_995995_11.stderr
parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 313e38acba438f3376afb697a9bda262 with result from task 131
parsl.dataflow.dflow DEBUG: Task 131 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a75fa8bb-210f-421f-be20-98f492a1819d_isrForCrosstalkSources_995995_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a75fa8bb-210f-421f-be20-98f492a1819d_isrForCrosstalkSources_995995_62.stderr
parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 30ee06b530bfc6ed1d94ae29ba3495a0 with result from task 135
parsl.dataflow.dflow DEBUG: Task 135 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/1c40addb-6103-4efd-a349-893351bfdb07_isrForCrosstalkSources_995848_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/1c40addb-6103-4efd-a349-893351bfdb07_isrForCrosstalkSources_995848_4.stderr
parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 13212acefcbdf4151ce740f8d3699ee2 with result from task 145
parsl.dataflow.dflow DEBUG: Task 145 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/78c08715-db15-4a81-a8c5-6cf058726709_isrForCrosstalkSources_995993_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/78c08715-db15-4a81-a8c5-6cf058726709_isrForCrosstalkSources_995993_23.stderr
2024-03-15 10:22:44 proc_lsst.multi:170 [INFO]  [multi] job_id 17064243
proc_lsst.multi INFO: [multi] job_id 17064243
2024-03-15 10:22:44 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-15 10:22:44 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:44 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064243
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064243
parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17064243
parsl.executors.status_handling INFO: Allocated block ID 55
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55
2024-03-15 10:22:44 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-55
2024-03-15 10:22:44 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:44 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:44 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:44 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:44 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-15 10:22:44 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:44 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=55 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to escience_a40
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e53a0631059547f06d596960737dd59e with result from task 143
parsl.dataflow.dflow DEBUG: Task 143 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/54e1e4c5-77a8-48f1-8100-2abd40d13b7b_isrForCrosstalkSources_995848_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/54e1e4c5-77a8-48f1-8100-2abd40d13b7b_isrForCrosstalkSources_995848_2.stderr
parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac12ac44899fb17a48e931df1607a7c with result from task 144
parsl.dataflow.dflow DEBUG: Task 144 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/34313f25-23c2-4162-80fb-1cc635bb77fd_isrForCrosstalkSources_995849_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/34313f25-23c2-4162-80fb-1cc635bb77fd_isrForCrosstalkSources_995849_11.stderr
parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d727b3ff53edb35e2f7b32817766eba with result from task 151
parsl.dataflow.dflow DEBUG: Task 151 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/04a8a54a-a903-42cc-bb01-b6526c1294f6_isrForCrosstalkSources_995848_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/04a8a54a-a903-42cc-bb01-b6526c1294f6_isrForCrosstalkSources_995848_13.stderr
parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 76eb6ddd162a6acbe1d50fb73d18325b with result from task 146
parsl.dataflow.dflow DEBUG: Task 146 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4b66d040-c6a0-4920-acc2-891468df0220_isrForCrosstalkSources_995994_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4b66d040-c6a0-4920-acc2-891468df0220_isrForCrosstalkSources_995994_8.stderr
2024-03-15 10:22:46 proc_lsst.multi:170 [INFO]  [multi] job_id 17064245
proc_lsst.multi INFO: [multi] job_id 17064245
2024-03-15 10:22:46 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:22:46 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:46 proc_lsst.multi:178 [INFO]  [multi] provider escience_a40 accepted submit and returned 17064245
proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17064245
parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17064245
parsl.executors.status_handling INFO: Allocated block ID 56
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56
2024-03-15 10:22:46 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-56
2024-03-15 10:22:46 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:46 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:46 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:46 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:22:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:46 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:46 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 0
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0
2024-03-15 10:22:46 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:46 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=56 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3b9c5973eb4f4de7f2a19e80cba9780e with result from task 156
parsl.dataflow.dflow DEBUG: Task 156 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4311957f-f902-49e3-9000-685e81557441_isrForCrosstalkSources_995994_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4311957f-f902-49e3-9000-685e81557441_isrForCrosstalkSources_995994_13.stderr
parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 42963926ccc70167f83182fbf076a33f with result from task 147
parsl.dataflow.dflow DEBUG: Task 147 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/69aff2a1-df2a-4528-8507-b464c07a91ea_isrForCrosstalkSources_995850_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/69aff2a1-df2a-4528-8507-b464c07a91ea_isrForCrosstalkSources_995850_23.stderr
parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 50199c8fdef787495a43e8a2f8426291 with result from task 148
parsl.dataflow.dflow DEBUG: Task 148 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b85bd10c-3aa5-443a-b1e4-1afb2d259947_isrForCrosstalkSources_995849_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b85bd10c-3aa5-443a-b1e4-1afb2d259947_isrForCrosstalkSources_995849_53.stderr
parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a006b23a21e92081ece2b8926fc9db26 with result from task 155
parsl.dataflow.dflow DEBUG: Task 155 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/af31ab4e-eda9-4635-94c9-6e2681225995_isrForCrosstalkSources_995850_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/af31ab4e-eda9-4635-94c9-6e2681225995_isrForCrosstalkSources_995850_45.stderr
parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b59f66db95e59d0414b0dcb84d7214d1 with result from task 149
parsl.dataflow.dflow DEBUG: Task 149 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/29a5a688-e31b-40f6-8292-96e07869cb90_isrForCrosstalkSources_995995_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/29a5a688-e31b-40f6-8292-96e07869cb90_isrForCrosstalkSources_995995_21.stderr
parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 2c35cdaed4c8c344e86204718ccdbc51 with result from task 152
parsl.dataflow.dflow DEBUG: Task 152 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3a27aad0-18a5-41fb-b057-65374dadf371_isrForCrosstalkSources_995995_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3a27aad0-18a5-41fb-b057-65374dadf371_isrForCrosstalkSources_995995_38.stderr
parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4afe3e83bb7ea24b691233a8b5fd93f0 with result from task 150
parsl.dataflow.dflow DEBUG: Task 150 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/14498e81-99d9-483e-b480-e5e07842e15e_isrForCrosstalkSources_995995_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/14498e81-99d9-483e-b480-e5e07842e15e_isrForCrosstalkSources_995995_23.stderr
parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1398d9f11fbe40adafa65a67d00c28e2 with result from task 157
parsl.dataflow.dflow DEBUG: Task 157 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a563cb65-1797-47b4-a9bb-8acf0df445a0_isrForCrosstalkSources_995848_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a563cb65-1797-47b4-a9bb-8acf0df445a0_isrForCrosstalkSources_995848_24.stderr
parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df954223226b19c36651cb85a45bc444 with result from task 109
parsl.dataflow.dflow DEBUG: Task 109 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c55b725f-0b03-4b5c-9833-c2c3859fdf31_isrForCrosstalkSources_995848_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c55b725f-0b03-4b5c-9833-c2c3859fdf31_isrForCrosstalkSources_995848_23.stderr
parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 04bbefb9cd07cf05eff71c96427016e4 with result from task 107
parsl.dataflow.dflow DEBUG: Task 107 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/bf904008-8ecd-47c7-b43e-dcb17cfd53d4_isrForCrosstalkSources_995848_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/bf904008-8ecd-47c7-b43e-dcb17cfd53d4_isrForCrosstalkSources_995848_27.stderr
parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 416aa34eb032c355a4ad808c6ef31e11 with result from task 108
parsl.dataflow.dflow DEBUG: Task 108 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/1090f174-b2af-43c6-895c-d568509c640e_isrForCrosstalkSources_995995_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/1090f174-b2af-43c6-895c-d568509c640e_isrForCrosstalkSources_995995_4.stderr
parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d3ec9f3e4effce1f510e2d88fe7a371e with result from task 154
parsl.dataflow.dflow DEBUG: Task 154 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/a80a5b04-305f-49e9-a593-80336c07b3ed_isrForCrosstalkSources_995850_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/a80a5b04-305f-49e9-a593-80336c07b3ed_isrForCrosstalkSources_995850_7.stderr
parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a19f5d1c7fb897dc44a4c82a74a652d7 with result from task 142
parsl.dataflow.dflow DEBUG: Task 142 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/3d7c468f-74c6-4aee-9014-4acda0ee8397_isrForCrosstalkSources_995850_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/3d7c468f-74c6-4aee-9014-4acda0ee8397_isrForCrosstalkSources_995850_22.stderr
parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b859681ed7270c76f4384aa55543346c with result from task 159
parsl.dataflow.dflow DEBUG: Task 159 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4c2f1d84-bfe5-4009-8753-2a1b49f54321_isrForCrosstalkSources_995850_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4c2f1d84-bfe5-4009-8753-2a1b49f54321_isrForCrosstalkSources_995850_39.stderr
parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a0c1455e3341d9d504388b7e9195e0b2 with result from task 138
parsl.dataflow.dflow DEBUG: Task 138 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/6781ee82-2327-44b1-8b39-ac983801e564_isrForCrosstalkSources_995995_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/6781ee82-2327-44b1-8b39-ac983801e564_isrForCrosstalkSources_995995_16.stderr
parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 31abcf216c3e94a46a56c029ae28744f with result from task 136
parsl.dataflow.dflow DEBUG: Task 136 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9cec309e-2bc6-4e9c-a7c6-ea83ec2ca5f4_isrForCrosstalkSources_995850_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/9cec309e-2bc6-4e9c-a7c6-ea83ec2ca5f4_isrForCrosstalkSources_995850_37.stderr
parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f776e768c6a837d1234e95e433a99a40 with result from task 153
parsl.dataflow.dflow DEBUG: Task 153 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/203b08ff-e3d9-4c76-99c5-e28fd3680fa3_isrForCrosstalkSources_995850_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/203b08ff-e3d9-4c76-99c5-e28fd3680fa3_isrForCrosstalkSources_995850_55.stderr
2024-03-15 10:22:48 proc_lsst.multi:170 [INFO]  [multi] job_id 17064259
proc_lsst.multi INFO: [multi] job_id 17064259
2024-03-15 10:22:48 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:48 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:48 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064259
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064259
parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17064259
parsl.executors.status_handling INFO: Allocated block ID 57
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57
2024-03-15 10:22:48 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-57
2024-03-15 10:22:48 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:48 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:48 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:48 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:22:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:48 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:48 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:48 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:48 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=57 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ff4e23022acfadde31937476c7eb4653 with result from task 161
parsl.dataflow.dflow DEBUG: Task 161 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/3bfc88e7-716c-4b4a-9353-daf10f496928_isrForCrosstalkSources_995850_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/3bfc88e7-716c-4b4a-9353-daf10f496928_isrForCrosstalkSources_995850_40.stderr
parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10d6c91ebdddc0a720d063e97cf737c4 with result from task 137
parsl.dataflow.dflow DEBUG: Task 137 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/1118cbed-f5da-4a6a-aaf3-995fa5b9d7aa_isrForCrosstalkSources_995850_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/1118cbed-f5da-4a6a-aaf3-995fa5b9d7aa_isrForCrosstalkSources_995850_54.stderr
parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 587432b979b67c832aee228f293405e2 with result from task 165
parsl.dataflow.dflow DEBUG: Task 165 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/a2dc27cc-4c19-46ff-bc23-e3995e5cbb83_isrForCrosstalkSources_995849_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/a2dc27cc-4c19-46ff-bc23-e3995e5cbb83_isrForCrosstalkSources_995849_39.stderr
parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ebf9f07b7568366013786cd0775b3331 with result from task 158
parsl.dataflow.dflow DEBUG: Task 158 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4d806bb5-6f32-4f0e-94b7-e25c8a064aed_isrForCrosstalkSources_995850_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4d806bb5-6f32-4f0e-94b7-e25c8a064aed_isrForCrosstalkSources_995850_62.stderr
parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 73b560b52e2795ea3a240ab4148bd123 with result from task 166
parsl.dataflow.dflow DEBUG: Task 166 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/5bc31b53-d1d8-41fb-88ca-b5c9740b33b7_isrForCrosstalkSources_995850_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/5bc31b53-d1d8-41fb-88ca-b5c9740b33b7_isrForCrosstalkSources_995850_27.stderr
parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 838c41a4a1a9bacc35f4f0d5a288d38e with result from task 168
parsl.dataflow.dflow DEBUG: Task 168 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9dfecef5-a6e1-411d-a065-90e1995cf295_isrForCrosstalkSources_995995_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9dfecef5-a6e1-411d-a065-90e1995cf295_isrForCrosstalkSources_995995_18.stderr
parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d20d37f0750d7f3ce755e6726d3a23cd with result from task 164
parsl.dataflow.dflow DEBUG: Task 164 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a0bd4609-7847-4943-ae95-8b019c6e9161_isrForCrosstalkSources_995995_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a0bd4609-7847-4943-ae95-8b019c6e9161_isrForCrosstalkSources_995995_45.stderr
parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eed0fd5e11a0555778651e8ac05ebc19 with result from task 163
parsl.dataflow.dflow DEBUG: Task 163 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/76cbffb4-3dfd-4a58-998f-d93e107030ad_isrForCrosstalkSources_995849_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/76cbffb4-3dfd-4a58-998f-d93e107030ad_isrForCrosstalkSources_995849_37.stderr
parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 93244afba29ddb05675aabc0b9e17c8a with result from task 139
parsl.dataflow.dflow DEBUG: Task 139 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/85555cb0-8fa6-4072-879e-25c6ab65b75d_isrForCrosstalkSources_995993_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/85555cb0-8fa6-4072-879e-25c6ab65b75d_isrForCrosstalkSources_995993_38.stderr
parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 40016074939a3ddca3d80f355269bd14 with result from task 167
parsl.dataflow.dflow DEBUG: Task 167 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/35243c84-6dec-44e2-a0d5-ede9d79d4c88_isrForCrosstalkSources_995849_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/35243c84-6dec-44e2-a0d5-ede9d79d4c88_isrForCrosstalkSources_995849_19.stderr
parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d7648629ba65e2c91285f8e3ef2b9d97 with result from task 170
parsl.dataflow.dflow DEBUG: Task 170 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/576844ba-f809-4cae-b46f-48483dfd362e_isrForCrosstalkSources_995993_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/576844ba-f809-4cae-b46f-48483dfd362e_isrForCrosstalkSources_995993_43.stderr
2024-03-15 10:22:50 proc_lsst.multi:170 [INFO]  [multi] job_id 17064262
proc_lsst.multi INFO: [multi] job_id 17064262
2024-03-15 10:22:50 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-15 10:22:50 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:50 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064262
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064262
parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17064262
parsl.executors.status_handling INFO: Allocated block ID 58
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58
2024-03-15 10:22:50 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-58
2024-03-15 10:22:50 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:50 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:50 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:50 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:22:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:50 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:50 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 2
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2
2024-03-15 10:22:50 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:50 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=58 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry af1323d6a3f0b224f1d6021f93357fb2 with result from task 106
parsl.dataflow.dflow DEBUG: Task 106 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1691016a-6af0-4f79-a9f9-600aa3e4ae64_isrForCrosstalkSources_995849_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/1691016a-6af0-4f79-a9f9-600aa3e4ae64_isrForCrosstalkSources_995849_48.stderr
parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8a72fe2afc72904bbe1c59cdd1cd791d with result from task 175
parsl.dataflow.dflow DEBUG: Task 175 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/0bf6e963-55b9-4784-9e84-550b5b2087b8_isrForCrosstalkSources_995849_23.stdout
parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/0bf6e963-55b9-4784-9e84-550b5b2087b8_isrForCrosstalkSources_995849_23.stderr
parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 927367ea91528c0586a2b372f18581cf with result from task 140
parsl.dataflow.dflow DEBUG: Task 140 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/61c36c85-df6f-403f-a71d-1ac094eba8fd_isrForCrosstalkSources_995995_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/61c36c85-df6f-403f-a71d-1ac094eba8fd_isrForCrosstalkSources_995995_48.stderr
parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 736ad5d017b617be2e0727db3a6ecdb9 with result from task 141
parsl.dataflow.dflow DEBUG: Task 141 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ee12d54a-18e8-4dc9-be79-8bda6f448c8e_isrForCrosstalkSources_995849_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ee12d54a-18e8-4dc9-be79-8bda6f448c8e_isrForCrosstalkSources_995849_60.stderr
parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e459ea3863d48ba0d27f167fdb84686d with result from task 171
parsl.dataflow.dflow DEBUG: Task 171 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/cbc89193-e5b9-42ad-9681-22101ba92622_isrForCrosstalkSources_995995_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/cbc89193-e5b9-42ad-9681-22101ba92622_isrForCrosstalkSources_995995_58.stderr
parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9e1142462a28b6ed958d433d932ae7d with result from task 174
parsl.dataflow.dflow DEBUG: Task 174 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e782728f-0cbf-4c20-8cb6-9af35ea53a9a_isrForCrosstalkSources_995849_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e782728f-0cbf-4c20-8cb6-9af35ea53a9a_isrForCrosstalkSources_995849_10.stderr
parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7a62219e8087d6239d756d1fa7293882 with result from task 173
parsl.dataflow.dflow DEBUG: Task 173 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/948f69ac-d1ae-461b-ba60-93b873326ad4_isrForCrosstalkSources_995849_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/948f69ac-d1ae-461b-ba60-93b873326ad4_isrForCrosstalkSources_995849_44.stderr
parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fe69475aea4bc2d99a33bdb4a250de2b with result from task 176
parsl.dataflow.dflow DEBUG: Task 176 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/db4d2c48-4195-404e-9f5c-6471d757f934_isrForCrosstalkSources_995850_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/db4d2c48-4195-404e-9f5c-6471d757f934_isrForCrosstalkSources_995850_57.stderr
parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5a04035c5653c967d8c472db44a01189 with result from task 169
parsl.dataflow.dflow DEBUG: Task 169 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/55af237e-4824-4dc7-84e3-7da75773cd68_isrForCrosstalkSources_995994_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/55af237e-4824-4dc7-84e3-7da75773cd68_isrForCrosstalkSources_995994_38.stderr
parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 39e89a61b7c545e317ce5031cdcf16a1 with result from task 172
parsl.dataflow.dflow DEBUG: Task 172 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e8800f65-5a87-45eb-84a7-780146d87c40_isrForCrosstalkSources_995994_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e8800f65-5a87-45eb-84a7-780146d87c40_isrForCrosstalkSources_995994_53.stderr
parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 284011833bb7456b935376ec81ec0967 with result from task 177
parsl.dataflow.dflow DEBUG: Task 177 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d8197fb0-19c1-44fa-a230-c1219dd9b3d4_isrForCrosstalkSources_995850_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d8197fb0-19c1-44fa-a230-c1219dd9b3d4_isrForCrosstalkSources_995850_35.stderr
2024-03-15 10:22:53 proc_lsst.multi:170 [INFO]  [multi] job_id 17064263
proc_lsst.multi INFO: [multi] job_id 17064263
2024-03-15 10:22:53 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-15 10:22:53 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:53 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064263
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064263
parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17064263
parsl.executors.status_handling INFO: Allocated block ID 59
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59
2024-03-15 10:22:53 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-59
2024-03-15 10:22:53 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:53 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:53 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:53 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:22:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:53 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:53 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 3
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3
2024-03-15 10:22:53 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:53 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=59 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c30ff55b00b697e12568524237e6b582 with result from task 160
parsl.dataflow.dflow DEBUG: Task 160 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/01fc0397-284b-4056-8300-9f44a15308ce_isrForCrosstalkSources_995994_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/01fc0397-284b-4056-8300-9f44a15308ce_isrForCrosstalkSources_995994_21.stderr
parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d798f2eb5010ae0d58b1611e67acb84d with result from task 178
parsl.dataflow.dflow DEBUG: Task 178 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/16ca6fb4-dcfb-46e3-ba4c-05ce14a92cbf_isrForCrosstalkSources_995850_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/16ca6fb4-dcfb-46e3-ba4c-05ce14a92cbf_isrForCrosstalkSources_995850_53.stderr
parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 38aa71ed6600e2f0819f5d0e0d8d1ec8 with result from task 179
parsl.dataflow.dflow DEBUG: Task 179 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/d79d5d89-e26a-439f-95cf-48b2b408c6b9_isrForCrosstalkSources_995848_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/d79d5d89-e26a-439f-95cf-48b2b408c6b9_isrForCrosstalkSources_995848_44.stderr
parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cb6a4f350a8647e6c5b94024f8a74931 with result from task 181
parsl.dataflow.dflow DEBUG: Task 181 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ab6cb5fb-332a-4c8a-abb3-06c276aa0770_isrForCrosstalkSources_995994_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ab6cb5fb-332a-4c8a-abb3-06c276aa0770_isrForCrosstalkSources_995994_27.stderr
parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5033405f7099523a3bc1b310ece9d5df with result from task 183
parsl.dataflow.dflow DEBUG: Task 183 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/f8d5b59e-4b6f-4b53-b34b-215e446d1929_isrForCrosstalkSources_995993_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/f8d5b59e-4b6f-4b53-b34b-215e446d1929_isrForCrosstalkSources_995993_3.stderr
parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 026c0e0024d09b3c7a3f36da297b56a0 with result from task 162
parsl.dataflow.dflow DEBUG: Task 162 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2a7d4f7c-38c6-42ee-84d1-10f746eec40f_isrForCrosstalkSources_995848_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2a7d4f7c-38c6-42ee-84d1-10f746eec40f_isrForCrosstalkSources_995848_61.stderr
parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 91201b42889f12a843ddfffe895f9e94 with result from task 180
parsl.dataflow.dflow DEBUG: Task 180 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9e4879cc-8b74-4015-83fa-ec24b1a7331b_isrForCrosstalkSources_995995_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9e4879cc-8b74-4015-83fa-ec24b1a7331b_isrForCrosstalkSources_995995_19.stderr
parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1d763fadd87d3064540d383dc4d3eb55 with result from task 182
parsl.dataflow.dflow DEBUG: Task 182 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e5171b5d-ed60-4d9f-a42f-902dd4bffebe_isrForCrosstalkSources_995849_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e5171b5d-ed60-4d9f-a42f-902dd4bffebe_isrForCrosstalkSources_995849_20.stderr
2024-03-15 10:22:54 proc_lsst.multi:170 [INFO]  [multi] job_id 17064264
proc_lsst.multi INFO: [multi] job_id 17064264
2024-03-15 10:22:54 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-15 10:22:54 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:54 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064264
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064264
parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17064264
parsl.executors.status_handling INFO: Allocated block ID 60
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60
2024-03-15 10:22:54 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-60
2024-03-15 10:22:54 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:54 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:54 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:54 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:22:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:54 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:54 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 4
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4
2024-03-15 10:22:54 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:54 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=60 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e97590027caefaa99bd9ec692aff7a23 with result from task 188
parsl.dataflow.dflow DEBUG: Task 188 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/09726804-eb8c-4c0c-a9b6-ea5719a84bc2_isrForCrosstalkSources_995849_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/09726804-eb8c-4c0c-a9b6-ea5719a84bc2_isrForCrosstalkSources_995849_8.stderr
parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5047255c14f8b920ad7b37c7de90bfd with result from task 184
parsl.dataflow.dflow DEBUG: Task 184 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/9e7f00e0-94a3-454f-b909-44316bf49161_isrForCrosstalkSources_995849_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/9e7f00e0-94a3-454f-b909-44316bf49161_isrForCrosstalkSources_995849_62.stderr
parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 331f254a91267daaa635eec2bf70a0d5 with result from task 186
parsl.dataflow.dflow DEBUG: Task 186 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/2cce04ed-b191-485d-a7e1-fb40f230191c_isrForCrosstalkSources_995994_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/2cce04ed-b191-485d-a7e1-fb40f230191c_isrForCrosstalkSources_995994_9.stderr
parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6c1b163121ac0a8dcb0a50a6a4c70bb3 with result from task 193
parsl.dataflow.dflow DEBUG: Task 193 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2b9947ee-b441-4c62-a70c-aafc70cd392d_isrForCrosstalkSources_995849_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2b9947ee-b441-4c62-a70c-aafc70cd392d_isrForCrosstalkSources_995849_33.stderr
parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 80ee55f1c194a9d115396238f4c31a3e with result from task 185
parsl.dataflow.dflow DEBUG: Task 185 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5cdc8b7a-cafc-405b-81b7-0f014044291e_isrForCrosstalkSources_995848_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5cdc8b7a-cafc-405b-81b7-0f014044291e_isrForCrosstalkSources_995848_40.stderr
parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6490334f1a0a5dd1742398dd27603a11 with result from task 187
parsl.dataflow.dflow DEBUG: Task 187 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/460c2cc6-6ec0-44f8-9414-bf2e6f1d5440_isrForCrosstalkSources_995994_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/460c2cc6-6ec0-44f8-9414-bf2e6f1d5440_isrForCrosstalkSources_995994_52.stderr
parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3dce88630aff583a171a35075717cc9b with result from task 189
parsl.dataflow.dflow DEBUG: Task 189 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e10eb275-377f-4fe5-86f1-8e38999474d4_isrForCrosstalkSources_995848_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e10eb275-377f-4fe5-86f1-8e38999474d4_isrForCrosstalkSources_995848_12.stderr
parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4922d3eab87c7bb8c695d3006cbc58dd with result from task 198
parsl.dataflow.dflow DEBUG: Task 198 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8e72e064-e252-4b4b-9f0b-767e218709ec_isrForCrosstalkSources_995849_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8e72e064-e252-4b4b-9f0b-767e218709ec_isrForCrosstalkSources_995849_1.stderr
2024-03-15 10:22:56 proc_lsst.multi:170 [INFO]  [multi] job_id 17064265
proc_lsst.multi INFO: [multi] job_id 17064265
2024-03-15 10:22:56 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:56 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:56 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064265
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064265
parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17064265
parsl.executors.status_handling INFO: Allocated block ID 61
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61
2024-03-15 10:22:56 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-61
2024-03-15 10:22:56 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:56 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:56 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:56 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:22:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:56 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:56 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:56 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:56 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=61 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3a7161731aa2bb8c8b12a72982bbb3d0 with result from task 199
parsl.dataflow.dflow DEBUG: Task 199 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/6253e8ca-76f7-4db5-8036-9af7352926a9_isrForCrosstalkSources_995849_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/6253e8ca-76f7-4db5-8036-9af7352926a9_isrForCrosstalkSources_995849_21.stderr
parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cda24867b379ce66fe37d05731be54d7 with result from task 194
parsl.dataflow.dflow DEBUG: Task 194 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4d2a2dbd-c8ca-41d5-8eac-782d0b3d9595_isrForCrosstalkSources_995994_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4d2a2dbd-c8ca-41d5-8eac-782d0b3d9595_isrForCrosstalkSources_995994_25.stderr
parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 757c4ac8e96d22245f546cca4527c107 with result from task 192
parsl.dataflow.dflow DEBUG: Task 192 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0deb3db9-733d-4a2e-a6e1-2bfcf9b222cc_isrForCrosstalkSources_995995_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0deb3db9-733d-4a2e-a6e1-2bfcf9b222cc_isrForCrosstalkSources_995995_3.stderr
parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry cd37f1e2ee84c447ccc4b4ebaa1dbb87 with result from task 195
parsl.dataflow.dflow DEBUG: Task 195 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0f1c9761-4eb4-4359-a3b3-592ec2e4f187_isrForCrosstalkSources_995848_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0f1c9761-4eb4-4359-a3b3-592ec2e4f187_isrForCrosstalkSources_995848_43.stderr
parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a5cba022186cf2e48b3494c529941f53 with result from task 197
parsl.dataflow.dflow DEBUG: Task 197 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/67d1d6a0-c843-45ca-ae94-dd70da994dfa_isrForCrosstalkSources_995993_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/67d1d6a0-c843-45ca-ae94-dd70da994dfa_isrForCrosstalkSources_995993_8.stderr
parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4c0717d68b836432597131af7c57c2bb with result from task 200
parsl.dataflow.dflow DEBUG: Task 200 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/b710789a-89e1-4f71-8254-1b9a107960bd_isrForCrosstalkSources_995995_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/b710789a-89e1-4f71-8254-1b9a107960bd_isrForCrosstalkSources_995995_28.stderr
parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f07b2df85382484dc93e799e6731de2b with result from task 208
parsl.dataflow.dflow DEBUG: Task 208 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/605b71a9-7f69-48ea-917a-26e3bdd5651d_isrForCrosstalkSources_995995_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/605b71a9-7f69-48ea-917a-26e3bdd5651d_isrForCrosstalkSources_995995_7.stderr
parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a5dfc40e0b4416bd385f8c5fffab85bc with result from task 190
parsl.dataflow.dflow DEBUG: Task 190 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5de5e1bb-b02c-4788-ba44-56f9517e0fdf_isrForCrosstalkSources_995848_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5de5e1bb-b02c-4788-ba44-56f9517e0fdf_isrForCrosstalkSources_995848_6.stderr
parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89d357f9945db447e4bb2953a2dbb3f3 with result from task 201
parsl.dataflow.dflow DEBUG: Task 201 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ec09aa3a-67e3-4819-ab3c-90e94c9a1b6e_isrForCrosstalkSources_995848_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ec09aa3a-67e3-4819-ab3c-90e94c9a1b6e_isrForCrosstalkSources_995848_62.stderr
parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3e81c11cd940c523ee0cb2ee7733e061 with result from task 191
parsl.dataflow.dflow DEBUG: Task 191 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/c7ef614e-ef2c-4a2e-b063-001e4152729d_isrForCrosstalkSources_995995_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/c7ef614e-ef2c-4a2e-b063-001e4152729d_isrForCrosstalkSources_995995_54.stderr
parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e5a3ee2cddb1612fc0df625bc37b00a7 with result from task 196
parsl.dataflow.dflow DEBUG: Task 196 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f1b53e87-056a-405d-9308-839b5ee0e132_isrForCrosstalkSources_995994_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f1b53e87-056a-405d-9308-839b5ee0e132_isrForCrosstalkSources_995994_60.stderr
parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ab2fb84de1218bb01b6d866d456a7d93 with result from task 203
parsl.dataflow.dflow DEBUG: Task 203 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/5b6eda7e-f323-44ce-aca8-6c106642f5ed_isrForCrosstalkSources_995995_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/5b6eda7e-f323-44ce-aca8-6c106642f5ed_isrForCrosstalkSources_995995_2.stderr
parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0eb5321908c8a441bc5317fb70d8624b with result from task 215
parsl.dataflow.dflow DEBUG: Task 215 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/08851989-460e-4893-bb6b-84f310a63006_isrForCrosstalkSources_995993_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/08851989-460e-4893-bb6b-84f310a63006_isrForCrosstalkSources_995993_1.stderr
2024-03-15 10:22:59 proc_lsst.multi:170 [INFO]  [multi] job_id 17064267
proc_lsst.multi INFO: [multi] job_id 17064267
2024-03-15 10:22:59 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-15 10:22:59 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:59 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064267
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064267
parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17064267
parsl.executors.status_handling INFO: Allocated block ID 62
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62
2024-03-15 10:22:59 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-62
2024-03-15 10:22:59 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:22:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:22:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:22:59 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:22:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:22:59 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:22:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:22:59 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:22:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:22:59 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:22:59 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 6
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6
2024-03-15 10:22:59 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:22:59 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=62 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 25acf659c91296365238fca524db5024 with result from task 206
parsl.dataflow.dflow DEBUG: Task 206 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/54abc801-c9a7-4c47-a3d5-4e4fa83e4a2c_isrForCrosstalkSources_995994_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/54abc801-c9a7-4c47-a3d5-4e4fa83e4a2c_isrForCrosstalkSources_995994_11.stderr
parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 61a58fcbda1d8d354489ed2b4efbe2d2 with result from task 211
parsl.dataflow.dflow DEBUG: Task 211 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/c7709f2f-3d37-4e44-a47a-52a90e9c6244_isrForCrosstalkSources_995994_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/c7709f2f-3d37-4e44-a47a-52a90e9c6244_isrForCrosstalkSources_995994_17.stderr
parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 256452ec810ee84f9c64930f35b1a98b with result from task 207
parsl.dataflow.dflow DEBUG: Task 207 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ebf381fd-4605-46df-866f-23d19d5b1d48_isrForCrosstalkSources_995849_35.stdout
parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ebf381fd-4605-46df-866f-23d19d5b1d48_isrForCrosstalkSources_995849_35.stderr
parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea3ce2df5d043e7f42b5f944df736e1 with result from task 202
parsl.dataflow.dflow DEBUG: Task 202 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/cb32a842-3ec8-4bfa-b22e-506a5ded5470_isrForCrosstalkSources_995993_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/cb32a842-3ec8-4bfa-b22e-506a5ded5470_isrForCrosstalkSources_995993_62.stderr
parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fdc322c9d0a450f063711f0ed43af766 with result from task 209
parsl.dataflow.dflow DEBUG: Task 209 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/d8650899-c629-4e3e-add5-bb96e989da65_isrForCrosstalkSources_995849_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/d8650899-c629-4e3e-add5-bb96e989da65_isrForCrosstalkSources_995849_59.stderr
parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dbc845d9a924fb0477564bac5e12eecb with result from task 212
parsl.dataflow.dflow DEBUG: Task 212 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/7792943f-5050-4394-ab8b-e293b47e8fd7_isrForCrosstalkSources_995994_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/7792943f-5050-4394-ab8b-e293b47e8fd7_isrForCrosstalkSources_995994_30.stderr
parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d41b10e276febbbd2aea7e231a73882c with result from task 222
parsl.dataflow.dflow DEBUG: Task 222 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/3d57a71d-4373-465f-bec7-10cefc4bf9bb_isrForCrosstalkSources_995993_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/3d57a71d-4373-465f-bec7-10cefc4bf9bb_isrForCrosstalkSources_995993_37.stderr
parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9a0923fa427e56be4766a7d0f7cdf891 with result from task 223
parsl.dataflow.dflow DEBUG: Task 223 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/234a3a1d-7478-437c-b3bc-0c46ee8330f0_isrForCrosstalkSources_995848_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/234a3a1d-7478-437c-b3bc-0c46ee8330f0_isrForCrosstalkSources_995848_39.stderr
parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bb4b35a0f12fc518d6ffe1bc77f10c65 with result from task 225
parsl.dataflow.dflow DEBUG: Task 225 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9ac42e83-9152-49d9-83d6-1f6d5fbf8df9_isrForCrosstalkSources_995993_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9ac42e83-9152-49d9-83d6-1f6d5fbf8df9_isrForCrosstalkSources_995993_13.stderr
parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4ccd48078fe182d5348612b3b4cbc1a9 with result from task 210
parsl.dataflow.dflow DEBUG: Task 210 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/310890b2-49ee-4b26-b70e-b20dd8341355_isrForCrosstalkSources_995848_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/310890b2-49ee-4b26-b70e-b20dd8341355_isrForCrosstalkSources_995848_28.stderr
parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33955a2ddc1e8028295f141b0dc62b0f with result from task 213
parsl.dataflow.dflow DEBUG: Task 213 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/ec671436-3995-40fd-a69a-3e1d2da96c74_isrForCrosstalkSources_995993_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/ec671436-3995-40fd-a69a-3e1d2da96c74_isrForCrosstalkSources_995993_25.stderr
parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 989cd4348aa253b90f04369eed42b8f4 with result from task 218
parsl.dataflow.dflow DEBUG: Task 218 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cb28a63a-aeb9-4fe9-90c9-869dc409d536_isrForCrosstalkSources_995994_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/cb28a63a-aeb9-4fe9-90c9-869dc409d536_isrForCrosstalkSources_995994_36.stderr
parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f212debadbdde6bb0889f12cf517126f with result from task 226
parsl.dataflow.dflow DEBUG: Task 226 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/5f7794b3-2824-4712-b5ea-124e7881c43f_isrForCrosstalkSources_995849_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/5f7794b3-2824-4712-b5ea-124e7881c43f_isrForCrosstalkSources_995849_18.stderr
parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7aba1b862762299c5665b47a02c80beb with result from task 216
parsl.dataflow.dflow DEBUG: Task 216 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/ac71fba8-eb7a-4b17-8676-c2dd13422bca_isrForCrosstalkSources_995995_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/ac71fba8-eb7a-4b17-8676-c2dd13422bca_isrForCrosstalkSources_995995_53.stderr
parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0333e6b694d6ed3862d8e58c0ae1e026 with result from task 214
parsl.dataflow.dflow DEBUG: Task 214 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a6b206e9-5210-4ffc-9a34-3cf746b98c8e_isrForCrosstalkSources_995848_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a6b206e9-5210-4ffc-9a34-3cf746b98c8e_isrForCrosstalkSources_995848_15.stderr
2024-03-15 10:23:01 proc_lsst.multi:170 [INFO]  [multi] job_id 17064268
proc_lsst.multi INFO: [multi] job_id 17064268
2024-03-15 10:23:01 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-15 10:23:01 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:01 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064268
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064268
parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17064268
parsl.executors.status_handling INFO: Allocated block ID 63
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63
2024-03-15 10:23:01 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-63
2024-03-15 10:23:01 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:01 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:01 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:01 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:01 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:01 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 7
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7
2024-03-15 10:23:01 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:01 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=63 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba10774e2bc37b97f1e6ea5ba26d74c7 with result from task 227
parsl.dataflow.dflow DEBUG: Task 227 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5b18fad0-1fc1-4f33-88fb-0537dff5b33f_isrForCrosstalkSources_995848_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5b18fad0-1fc1-4f33-88fb-0537dff5b33f_isrForCrosstalkSources_995848_37.stderr
parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9f4a068cb31055c398780249f14972b with result from task 228
parsl.dataflow.dflow DEBUG: Task 228 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c020f498-99eb-42b5-914a-f5d28fa3071c_isrForCrosstalkSources_995849_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c020f498-99eb-42b5-914a-f5d28fa3071c_isrForCrosstalkSources_995849_31.stderr
parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9ee1d59e229622c05feb5db23a39da59 with result from task 234
parsl.dataflow.dflow DEBUG: Task 234 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e3532a13-d6f7-4d0c-892c-259c9f18947f_isrForCrosstalkSources_995993_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e3532a13-d6f7-4d0c-892c-259c9f18947f_isrForCrosstalkSources_995993_40.stderr
parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36a81e9f9ca94deb9b33b0be1918a327 with result from task 229
parsl.dataflow.dflow DEBUG: Task 229 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/3659651f-e170-4c3a-aeed-4736992c9321_isrForCrosstalkSources_995994_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/3659651f-e170-4c3a-aeed-4736992c9321_isrForCrosstalkSources_995994_54.stderr
parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5d2d3d905f7d54715b2e8d5a845c4678 with result from task 230
parsl.dataflow.dflow DEBUG: Task 230 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/52d34415-4743-4a47-9954-cf0df8e402b4_isrForCrosstalkSources_995850_21.stdout
parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/52d34415-4743-4a47-9954-cf0df8e402b4_isrForCrosstalkSources_995850_21.stderr
parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f746df3be05ea1a2ae8703b0cf3eaeae with result from task 231
parsl.dataflow.dflow DEBUG: Task 231 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6b0d42fd-a500-493d-8405-6052f6831302_isrForCrosstalkSources_995993_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6b0d42fd-a500-493d-8405-6052f6831302_isrForCrosstalkSources_995993_39.stderr
parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89db020f5b170f50839d538aedb0cadb with result from task 232
parsl.dataflow.dflow DEBUG: Task 232 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f481c886-4b79-42fc-83c0-420d24407ce8_isrForCrosstalkSources_995995_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f481c886-4b79-42fc-83c0-420d24407ce8_isrForCrosstalkSources_995995_6.stderr
2024-03-15 10:23:03 proc_lsst.multi:170 [INFO]  [multi] job_id 17064273
proc_lsst.multi INFO: [multi] job_id 17064273
2024-03-15 10:23:03 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-15 10:23:03 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:03 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064273
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064273
parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17064273
parsl.executors.status_handling INFO: Allocated block ID 64
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64
2024-03-15 10:23:03 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-64
2024-03-15 10:23:03 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:03 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:03 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:03 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:03 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:03 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 8
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8
2024-03-15 10:23:03 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:03 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=64 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba8f9affc63608da700392da6c08794e with result from task 235
parsl.dataflow.dflow DEBUG: Task 235 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/744d950e-cc8c-427a-9d7c-7c78189d9931_isrForCrosstalkSources_995994_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/744d950e-cc8c-427a-9d7c-7c78189d9931_isrForCrosstalkSources_995994_1.stderr
parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d46bb39edb9d7a0eae7a75e16321b7e1 with result from task 241
parsl.dataflow.dflow DEBUG: Task 241 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/011feedf-41b9-4b81-8787-26f35696212e_isrForCrosstalkSources_995993_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/011feedf-41b9-4b81-8787-26f35696212e_isrForCrosstalkSources_995993_32.stderr
parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0c02f7a69ae020f0894b91b8a6dbd7d3 with result from task 205
parsl.dataflow.dflow DEBUG: Task 205 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c6c449f4-24e3-4cf7-9bf2-ecc11097f489_isrForCrosstalkSources_995848_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c6c449f4-24e3-4cf7-9bf2-ecc11097f489_isrForCrosstalkSources_995848_30.stderr
parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e9891956f226d6f8cf1a9c31bbb66136 with result from task 236
parsl.dataflow.dflow DEBUG: Task 236 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8f254323-c096-4408-b996-5d06a5a23cbb_isrForCrosstalkSources_995849_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8f254323-c096-4408-b996-5d06a5a23cbb_isrForCrosstalkSources_995849_45.stderr
parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 23a9c512b7bf18835990a4916384450c with result from task 237
parsl.dataflow.dflow DEBUG: Task 237 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/243cba2f-6c57-4a3b-a469-dd8104d5f340_isrForCrosstalkSources_995849_40.stdout
parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/243cba2f-6c57-4a3b-a469-dd8104d5f340_isrForCrosstalkSources_995849_40.stderr
parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 25ef2d7b3350a3863988778dc344c9f0 with result from task 204
parsl.dataflow.dflow DEBUG: Task 204 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/4ec43633-4d0f-4132-a766-1447426002d2_isrForCrosstalkSources_995848_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/4ec43633-4d0f-4132-a766-1447426002d2_isrForCrosstalkSources_995848_52.stderr
parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0fe1502b73db89abf1d55e312d7519e7 with result from task 244
parsl.dataflow.dflow DEBUG: Task 244 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/db77a0fc-3b2f-4f02-803f-03f10f733501_isrForCrosstalkSources_995994_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/db77a0fc-3b2f-4f02-803f-03f10f733501_isrForCrosstalkSources_995994_26.stderr
parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 323667d478882400095b1757fcb57b3e with result from task 224
parsl.dataflow.dflow DEBUG: Task 224 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/465138b3-e084-457d-8ebf-976f04cf00e2_isrForCrosstalkSources_995993_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/465138b3-e084-457d-8ebf-976f04cf00e2_isrForCrosstalkSources_995993_2.stderr
parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c986fdd4ce9bc0b951047617fd9faea with result from task 233
parsl.dataflow.dflow DEBUG: Task 233 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a3b2659b-e1a2-44a1-9119-ce9366768fdf_isrForCrosstalkSources_995848_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/a3b2659b-e1a2-44a1-9119-ce9366768fdf_isrForCrosstalkSources_995848_34.stderr
parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 24904d974d17380082c232e15cb688ce with result from task 242
parsl.dataflow.dflow DEBUG: Task 242 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d5fe47d3-9435-4f60-be8c-00921a22ba09_isrForCrosstalkSources_995850_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d5fe47d3-9435-4f60-be8c-00921a22ba09_isrForCrosstalkSources_995850_13.stderr
parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 85732cd9e1e127890ced450dc936fd9e with result from task 248
parsl.dataflow.dflow DEBUG: Task 248 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8dfaed11-9b50-48e4-83bf-f7bf9fafee16_isrForCrosstalkSources_995850_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8dfaed11-9b50-48e4-83bf-f7bf9fafee16_isrForCrosstalkSources_995850_5.stderr
parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f2081b87a18b862ca652948785a649cc with result from task 245
parsl.dataflow.dflow DEBUG: Task 245 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/6660b4c4-6975-464b-a87c-472a856f42d9_isrForCrosstalkSources_995849_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/6660b4c4-6975-464b-a87c-472a856f42d9_isrForCrosstalkSources_995849_5.stderr
2024-03-15 10:23:05 proc_lsst.multi:170 [INFO]  [multi] job_id 17064279
proc_lsst.multi INFO: [multi] job_id 17064279
2024-03-15 10:23:05 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-15 10:23:05 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:05 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064279
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064279
parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17064279
parsl.executors.status_handling INFO: Allocated block ID 65
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65
2024-03-15 10:23:05 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-65
2024-03-15 10:23:05 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:05 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:05 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:05 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:05 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:05 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 9
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9
2024-03-15 10:23:05 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:05 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=65 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry bbbd3bb20fb2887283dc7cc03fd31db3 with result from task 239
parsl.dataflow.dflow DEBUG: Task 239 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/239eb505-8d47-477b-962d-3f4db2f91951_isrForCrosstalkSources_995849_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/239eb505-8d47-477b-962d-3f4db2f91951_isrForCrosstalkSources_995849_42.stderr
parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b9ac5a3f56117df124a1ea09bd4e1f6 with result from task 240
parsl.dataflow.dflow DEBUG: Task 240 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ed07c81f-ea3c-401b-bed6-ee5db21fa41e_isrForCrosstalkSources_995994_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ed07c81f-ea3c-401b-bed6-ee5db21fa41e_isrForCrosstalkSources_995994_50.stderr
parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5813111deace37c70b5f860c513caf10 with result from task 243
parsl.dataflow.dflow DEBUG: Task 243 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b11daaaa-6e74-49e0-8bf5-b2747d121e14_isrForCrosstalkSources_995849_6.stdout
parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b11daaaa-6e74-49e0-8bf5-b2747d121e14_isrForCrosstalkSources_995849_6.stderr
parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d62d19c3216504475342ba584d55a36f with result from task 246
parsl.dataflow.dflow DEBUG: Task 246 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/028036fc-d90f-43af-b8e3-094c2269d7d8_isrForCrosstalkSources_995849_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/028036fc-d90f-43af-b8e3-094c2269d7d8_isrForCrosstalkSources_995849_38.stderr
parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 311bbe79b2cc81f6b49fa406889aa830 with result from task 247
parsl.dataflow.dflow DEBUG: Task 247 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/0ad2f158-3fff-4117-bce7-87c0244b3c9c_isrForCrosstalkSources_995850_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/0ad2f158-3fff-4117-bce7-87c0244b3c9c_isrForCrosstalkSources_995850_26.stderr
parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 51173600ee53321d7e4370d3bcdedb95 with result from task 254
parsl.dataflow.dflow DEBUG: Task 254 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e21df543-19d9-4865-9447-9cb4a0236a17_isrForCrosstalkSources_995993_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e21df543-19d9-4865-9447-9cb4a0236a17_isrForCrosstalkSources_995993_15.stderr
2024-03-15 10:23:07 proc_lsst.multi:170 [INFO]  [multi] job_id 17064280
proc_lsst.multi INFO: [multi] job_id 17064280
2024-03-15 10:23:07 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-15 10:23:07 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:07 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064280
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064280
parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17064280
parsl.executors.status_handling INFO: Allocated block ID 66
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66
2024-03-15 10:23:07 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-66
2024-03-15 10:23:07 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:07 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:07 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:07 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:07 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:07 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 10
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10
2024-03-15 10:23:07 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:07 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=66 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 40f7182b00f0714d931014247e8a367a with result from task 255
parsl.dataflow.dflow DEBUG: Task 255 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/75552959-7bfe-4228-a1df-ded6ab3ed41f_isrForCrosstalkSources_995848_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/75552959-7bfe-4228-a1df-ded6ab3ed41f_isrForCrosstalkSources_995848_22.stderr
parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5946903fdc416498cceaf4f26c8a7ef2 with result from task 257
parsl.dataflow.dflow DEBUG: Task 257 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2adffdf3-ea8d-4ca9-b60b-ca3e9910f835_isrForCrosstalkSources_995848_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2adffdf3-ea8d-4ca9-b60b-ca3e9910f835_isrForCrosstalkSources_995848_42.stderr
parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c3e7b8a37b18b5bc23a1197d7dc74cc5 with result from task 261
parsl.dataflow.dflow DEBUG: Task 261 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/aec48142-fb39-4b65-a506-72810d3eda5e_isrForCrosstalkSources_995993_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/aec48142-fb39-4b65-a506-72810d3eda5e_isrForCrosstalkSources_995993_17.stderr
parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 718afab05378b7c209df3de69d975af3 with result from task 251
parsl.dataflow.dflow DEBUG: Task 251 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ca165be4-2355-4eaf-906e-f08541f227b6_isrForCrosstalkSources_995848_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ca165be4-2355-4eaf-906e-f08541f227b6_isrForCrosstalkSources_995848_33.stderr
parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed945e3ee2eba9ec33448d87e1dce692 with result from task 250
parsl.dataflow.dflow DEBUG: Task 250 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c57b43cb-9e30-4594-9e67-5781f564a08d_isrForCrosstalkSources_995848_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c57b43cb-9e30-4594-9e67-5781f564a08d_isrForCrosstalkSources_995848_20.stderr
parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry db4079ce6ef152e395142b56661170b7 with result from task 253
parsl.dataflow.dflow DEBUG: Task 253 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/f1dd7629-4123-4363-b823-164966707a16_isrForCrosstalkSources_995849_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/f1dd7629-4123-4363-b823-164966707a16_isrForCrosstalkSources_995849_24.stderr
parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c9fdf741ea4b5de0873d620ff91d5b39 with result from task 256
parsl.dataflow.dflow DEBUG: Task 256 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/84b57c14-6127-4750-a79e-49cddfcd0ae0_isrForCrosstalkSources_995848_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/84b57c14-6127-4750-a79e-49cddfcd0ae0_isrForCrosstalkSources_995848_49.stderr
parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 623e8bed796ac479f241ce67bdf41e30 with result from task 252
parsl.dataflow.dflow DEBUG: Task 252 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/62c87c26-4905-4152-aa1a-607e3b6f2abd_isrForCrosstalkSources_995848_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/62c87c26-4905-4152-aa1a-607e3b6f2abd_isrForCrosstalkSources_995848_47.stderr
parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 47f52dd1b03ffdda7fb122cda5b5158f with result from task 265
parsl.dataflow.dflow DEBUG: Task 265 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/13018970-48af-4bc0-924c-e3de3d9f07b8_isrForCrosstalkSources_995850_18.stdout
parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/13018970-48af-4bc0-924c-e3de3d9f07b8_isrForCrosstalkSources_995850_18.stderr
parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry df629c9a0b9bc25430faf98bfa0d1546 with result from task 260
parsl.dataflow.dflow DEBUG: Task 260 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/57aa5aae-f2e1-46eb-a4c0-a0e08f138e40_isrForCrosstalkSources_995848_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/57aa5aae-f2e1-46eb-a4c0-a0e08f138e40_isrForCrosstalkSources_995848_5.stderr
parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10b342beac1d1433e17b71c51f513c67 with result from task 270
parsl.dataflow.dflow DEBUG: Task 270 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/170001b8-1fd0-40fe-b76c-37cd06138dd4_isrForCrosstalkSources_995849_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/170001b8-1fd0-40fe-b76c-37cd06138dd4_isrForCrosstalkSources_995849_25.stderr
2024-03-15 10:23:09 proc_lsst.multi:170 [INFO]  [multi] job_id 17064281
proc_lsst.multi INFO: [multi] job_id 17064281
2024-03-15 10:23:09 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-15 10:23:09 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:09 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064281
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064281
parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17064281
parsl.executors.status_handling INFO: Allocated block ID 67
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67
2024-03-15 10:23:09 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-67
2024-03-15 10:23:09 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:09 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:09 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:09 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:09 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:09 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 11
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11
2024-03-15 10:23:09 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:09 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=67 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8cbd0342829824b6bb619fa959d29dfd with result from task 269
parsl.dataflow.dflow DEBUG: Task 269 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e3d2365f-71ca-4592-bd43-0671932ec901_isrForCrosstalkSources_995849_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/e3d2365f-71ca-4592-bd43-0671932ec901_isrForCrosstalkSources_995849_17.stderr
parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0b34624b9c57b196f7c4bc99532e9778 with result from task 274
parsl.dataflow.dflow DEBUG: Task 274 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2d105d32-b39c-4124-a700-76c4e3dee4ad_isrForCrosstalkSources_995849_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/2d105d32-b39c-4124-a700-76c4e3dee4ad_isrForCrosstalkSources_995849_51.stderr
parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c141d75598285031a247c97401f7517a with result from task 258
parsl.dataflow.dflow DEBUG: Task 258 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/677ef656-729c-4ddb-a752-c901138f06a0_isrForCrosstalkSources_995850_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/677ef656-729c-4ddb-a752-c901138f06a0_isrForCrosstalkSources_995850_24.stderr
parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f52c77e52f498454b0a190411f0170a with result from task 259
parsl.dataflow.dflow DEBUG: Task 259 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/7645d4d6-eb9b-4c99-8b0b-0bedea2028b8_isrForCrosstalkSources_995849_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/7645d4d6-eb9b-4c99-8b0b-0bedea2028b8_isrForCrosstalkSources_995849_30.stderr
parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e1b55ca66d832def134335cd920fd745 with result from task 264
parsl.dataflow.dflow DEBUG: Task 264 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/6ba23c3f-fc59-40eb-bf41-f70898334e46_isrForCrosstalkSources_995850_33.stdout
parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/6ba23c3f-fc59-40eb-bf41-f70898334e46_isrForCrosstalkSources_995850_33.stderr
parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 94b2cfe1e9f2ebf16e6bb715e555e015 with result from task 263
parsl.dataflow.dflow DEBUG: Task 263 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/51dc6a86-cb6b-4a83-ad7d-26d8163ed188_isrForCrosstalkSources_995849_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/51dc6a86-cb6b-4a83-ad7d-26d8163ed188_isrForCrosstalkSources_995849_58.stderr
parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 13eba9fb520cf45cb0817308e0a0cabb with result from task 266
parsl.dataflow.dflow DEBUG: Task 266 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5ab11120-89b7-4aa6-84be-809fef29607b_isrForCrosstalkSources_995994_22.stdout
parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5ab11120-89b7-4aa6-84be-809fef29607b_isrForCrosstalkSources_995994_22.stderr
parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4faeb6a032d773e93d316406504ebd69 with result from task 267
parsl.dataflow.dflow DEBUG: Task 267 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dbd14417-6d56-4d75-a4c7-b472e50919e7_isrForCrosstalkSources_995848_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/dbd14417-6d56-4d75-a4c7-b472e50919e7_isrForCrosstalkSources_995848_57.stderr
parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 917c9020890c5b619124d95f2b0a680b with result from task 268
parsl.dataflow.dflow DEBUG: Task 268 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/80700b4d-1666-4264-83fd-18ec3ca68289_isrForCrosstalkSources_995993_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/80700b4d-1666-4264-83fd-18ec3ca68289_isrForCrosstalkSources_995993_58.stderr
parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10330a3f2b36283752f8191fd5f95757 with result from task 249
parsl.dataflow.dflow DEBUG: Task 249 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/9340fe65-0f5b-48ad-85ba-f878c7647fd1_isrForCrosstalkSources_995994_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/9340fe65-0f5b-48ad-85ba-f878c7647fd1_isrForCrosstalkSources_995994_48.stderr
parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 05812054145f2576b51912e8aa3bfcf7 with result from task 262
parsl.dataflow.dflow DEBUG: Task 262 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/7a79c12b-4310-4960-9894-3dd2aaa79a95_isrForCrosstalkSources_995850_30.stdout
parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/7a79c12b-4310-4960-9894-3dd2aaa79a95_isrForCrosstalkSources_995850_30.stderr
parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 89db8f470efc622c0c90f7cc5c01ef22 with result from task 220
parsl.dataflow.dflow DEBUG: Task 220 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d3c600ea-a05a-4a29-bcfd-9d541163e33b_isrForCrosstalkSources_995850_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d3c600ea-a05a-4a29-bcfd-9d541163e33b_isrForCrosstalkSources_995850_4.stderr
parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58a9926ebb62597b6c5aed4838bb7ade with result from task 217
parsl.dataflow.dflow DEBUG: Task 217 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4899b1cf-207a-4382-9665-5a11b4f2988f_isrForCrosstalkSources_995850_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4899b1cf-207a-4382-9665-5a11b4f2988f_isrForCrosstalkSources_995850_8.stderr
parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 072d229ada02b292b1ab55abb62907f2 with result from task 221
parsl.dataflow.dflow DEBUG: Task 221 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8ad84c60-0025-4607-80be-1537336fec6a_isrForCrosstalkSources_995850_9.stdout
parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/8ad84c60-0025-4607-80be-1537336fec6a_isrForCrosstalkSources_995850_9.stderr
parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4bb204f1a93faeee7ef37c1625ea6ae5 with result from task 271
parsl.dataflow.dflow DEBUG: Task 271 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/f8d9f080-12dc-4bce-be8d-c858a37d271d_isrForCrosstalkSources_995848_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/f8d9f080-12dc-4bce-be8d-c858a37d271d_isrForCrosstalkSources_995848_32.stderr
parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78bd6dd620c9ac59d31c1d46cdd9d5b8 with result from task 273
parsl.dataflow.dflow DEBUG: Task 273 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/23a9a687-760c-4628-96c1-b8a7dc4c4c8f_isrForCrosstalkSources_995850_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/23a9a687-760c-4628-96c1-b8a7dc4c4c8f_isrForCrosstalkSources_995850_29.stderr
parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63228f76f8e67e7d43ef70fd9bab8ab6 with result from task 219
parsl.dataflow.dflow DEBUG: Task 219 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/870d8c29-c66c-46ed-8f84-784fe7d2b52b_isrForCrosstalkSources_995848_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/870d8c29-c66c-46ed-8f84-784fe7d2b52b_isrForCrosstalkSources_995848_10.stderr
parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 762775f2291dff0e69f05a1702e177c7 with result from task 238
parsl.dataflow.dflow DEBUG: Task 238 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/a7786c6d-637a-49d8-952c-370325f36647_isrForCrosstalkSources_995993_53.stdout
parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/a7786c6d-637a-49d8-952c-370325f36647_isrForCrosstalkSources_995993_53.stderr
2024-03-15 10:23:11 proc_lsst.multi:170 [INFO]  [multi] job_id 17064282
proc_lsst.multi INFO: [multi] job_id 17064282
2024-03-15 10:23:11 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-15 10:23:11 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:11 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064282
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064282
parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17064282
parsl.executors.status_handling INFO: Allocated block ID 68
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68
2024-03-15 10:23:11 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-68
2024-03-15 10:23:11 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:11 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:11 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:11 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:11 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:11 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 12
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12
2024-03-15 10:23:11 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:11 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=68 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 53361e152da5e7106d97b66b6a2b48b1 with result from task 279
parsl.dataflow.dflow DEBUG: Task 279 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b71a53cd-270d-4323-8f04-2788c03aaca0_isrForCrosstalkSources_995849_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b71a53cd-270d-4323-8f04-2788c03aaca0_isrForCrosstalkSources_995849_16.stderr
parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5cca11204331925db6e222559cda6cf8 with result from task 272
parsl.dataflow.dflow DEBUG: Task 272 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/846bfe47-17d6-4a86-a0fa-69242dd63180_isrForCrosstalkSources_995993_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/846bfe47-17d6-4a86-a0fa-69242dd63180_isrForCrosstalkSources_995993_54.stderr
parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6b90e60cbad4f6c27505094fb6a91923 with result from task 275
parsl.dataflow.dflow DEBUG: Task 275 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/56907ed8-22dc-44c9-a341-a9c143a92f7c_isrForCrosstalkSources_995848_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/56907ed8-22dc-44c9-a341-a9c143a92f7c_isrForCrosstalkSources_995848_29.stderr
parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9462425360779d85a6ce775a39af1172 with result from task 276
parsl.dataflow.dflow DEBUG: Task 276 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8cb2490f-a62b-4614-9892-8f5c4daf3d18_isrForCrosstalkSources_995849_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/8cb2490f-a62b-4614-9892-8f5c4daf3d18_isrForCrosstalkSources_995849_3.stderr
2024-03-15 10:23:12 proc_lsst.multi:170 [INFO]  [multi] job_id 17064285
proc_lsst.multi INFO: [multi] job_id 17064285
2024-03-15 10:23:12 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-15 10:23:12 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:12 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064285
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064285
parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17064285
parsl.executors.status_handling INFO: Allocated block ID 69
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69
2024-03-15 10:23:12 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-69
2024-03-15 10:23:12 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:12 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:12 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:12 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:12 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:12 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 13
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13
2024-03-15 10:23:12 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:12 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=69 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 43bb8ae330f3d0fea1b5356c37bf18d4 with result from task 277
parsl.dataflow.dflow DEBUG: Task 277 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/385851eb-110c-41ab-bef1-6edd42046876_isrForCrosstalkSources_995850_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/385851eb-110c-41ab-bef1-6edd42046876_isrForCrosstalkSources_995850_50.stderr
parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 73d9332620b4072d3b26bf052fa48a2c with result from task 280
parsl.dataflow.dflow DEBUG: Task 280 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/507bef1e-a4f0-4b8a-87a8-f33d8adda184_isrForCrosstalkSources_995994_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/507bef1e-a4f0-4b8a-87a8-f33d8adda184_isrForCrosstalkSources_995994_32.stderr
parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ed468954f7953862fed50eeba4d7550a with result from task 294
parsl.dataflow.dflow DEBUG: Task 294 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ca062d6e-c137-422e-83d9-e408828f5c96_isrForCrosstalkSources_995994_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/ca062d6e-c137-422e-83d9-e408828f5c96_isrForCrosstalkSources_995994_56.stderr
parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f7cd543dfb2f81eaacef3b45e2afadf3 with result from task 293
parsl.dataflow.dflow DEBUG: Task 293 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e5637648-22bc-422a-8926-22be81e5fcad_isrForCrosstalkSources_995848_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/e5637648-22bc-422a-8926-22be81e5fcad_isrForCrosstalkSources_995848_25.stderr
parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 28609ce5cec0fe122022cdd08f76823a with result from task 282
parsl.dataflow.dflow DEBUG: Task 282 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/217f690d-228a-4a98-9b92-1e89ee67d133_isrForCrosstalkSources_995848_8.stdout
parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/217f690d-228a-4a98-9b92-1e89ee67d133_isrForCrosstalkSources_995848_8.stderr
parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0dbce6f2612b17ccc16070cd295a9325 with result from task 283
parsl.dataflow.dflow DEBUG: Task 283 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9d90ecc8-5ab7-460d-a65f-6b813a9af576_isrForCrosstalkSources_995995_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/9d90ecc8-5ab7-460d-a65f-6b813a9af576_isrForCrosstalkSources_995995_47.stderr
parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c74bc3768f673d9eb99cecbfd54b72af with result from task 291
parsl.dataflow.dflow DEBUG: Task 291 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e5755a9e-8cb3-46b1-9fae-3744b12d7865_isrForCrosstalkSources_995994_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/e5755a9e-8cb3-46b1-9fae-3744b12d7865_isrForCrosstalkSources_995994_39.stderr
parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a72a316e4daaf2acf6c0ba14a170a6f6 with result from task 278
parsl.dataflow.dflow DEBUG: Task 278 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/21f2db4e-d901-4766-9942-5787e4cdf2a8_isrForCrosstalkSources_995995_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/21f2db4e-d901-4766-9942-5787e4cdf2a8_isrForCrosstalkSources_995995_29.stderr
parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 78b49792412bc57afe5f411fee489445 with result from task 289
parsl.dataflow.dflow DEBUG: Task 289 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/a6e1556f-295b-411d-a0a7-b1d15e4a3508_isrForCrosstalkSources_995993_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/a6e1556f-295b-411d-a0a7-b1d15e4a3508_isrForCrosstalkSources_995993_41.stderr
parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 33965be499e8cad9e10c295cf97b47f3 with result from task 281
parsl.dataflow.dflow DEBUG: Task 281 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/189b01fa-aa66-4d13-b34a-1038ec230d57_isrForCrosstalkSources_995995_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/189b01fa-aa66-4d13-b34a-1038ec230d57_isrForCrosstalkSources_995995_59.stderr
parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1dff35147d2172fac86b5502af60e317 with result from task 284
parsl.dataflow.dflow DEBUG: Task 284 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2fdc780f-23b9-44cf-83e6-2f90a72d8f86_isrForCrosstalkSources_995848_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/2fdc780f-23b9-44cf-83e6-2f90a72d8f86_isrForCrosstalkSources_995848_48.stderr
2024-03-15 10:23:14 proc_lsst.multi:170 [INFO]  [multi] job_id 17064286
proc_lsst.multi INFO: [multi] job_id 17064286
2024-03-15 10:23:14 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-15 10:23:14 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:14 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064286
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064286
parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17064286
parsl.executors.status_handling INFO: Allocated block ID 70
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70
2024-03-15 10:23:14 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-70
2024-03-15 10:23:14 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:14 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:14 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:14 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:14 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:14 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 14
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14
2024-03-15 10:23:14 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:14 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=70 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5fabd5e0e1395764ea246a693e8397f9 with result from task 288
parsl.dataflow.dflow DEBUG: Task 288 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/c9a9dc66-f816-4405-a9d8-7559d6a0fdb5_isrForCrosstalkSources_995995_14.stdout
parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/c9a9dc66-f816-4405-a9d8-7559d6a0fdb5_isrForCrosstalkSources_995995_14.stderr
parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1aab462463e86d630a2b1fe82d03c155 with result from task 285
parsl.dataflow.dflow DEBUG: Task 285 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/20475285-07c4-4176-82fb-9603bab2ea55_isrForCrosstalkSources_995848_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/20475285-07c4-4176-82fb-9603bab2ea55_isrForCrosstalkSources_995848_7.stderr
parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 530bae6afc916a94d0ab8e3ec56fa448 with result from task 295
parsl.dataflow.dflow DEBUG: Task 295 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/25034c20-6a03-4990-a96e-3db5bfb189b5_isrForCrosstalkSources_995995_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/25034c20-6a03-4990-a96e-3db5bfb189b5_isrForCrosstalkSources_995995_1.stderr
parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b8d52ac46c6c6b42462ad77e3c401c46 with result from task 286
parsl.dataflow.dflow DEBUG: Task 286 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f1fcddd5-d5d3-458e-a412-263a37be3160_isrForCrosstalkSources_995995_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f1fcddd5-d5d3-458e-a412-263a37be3160_isrForCrosstalkSources_995995_36.stderr
parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry caaa20046de055c0258fbbaa36ba0695 with result from task 287
parsl.dataflow.dflow DEBUG: Task 287 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/460cfe00-7b2c-40bb-9544-0008d2620026_isrForCrosstalkSources_995850_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/460cfe00-7b2c-40bb-9544-0008d2620026_isrForCrosstalkSources_995850_20.stderr
parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d793638d2b7513ac6f1c63f9fb9e1ca0 with result from task 290
parsl.dataflow.dflow DEBUG: Task 290 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c7d2f9db-f00c-4852-a1a8-81d4a18e135f_isrForCrosstalkSources_995849_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/c7d2f9db-f00c-4852-a1a8-81d4a18e135f_isrForCrosstalkSources_995849_52.stderr
2024-03-15 10:23:16 proc_lsst.multi:170 [INFO]  [multi] job_id 17064287
proc_lsst.multi INFO: [multi] job_id 17064287
2024-03-15 10:23:16 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-15 10:23:16 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:16 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064287
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064287
parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17064287
parsl.executors.status_handling INFO: Allocated block ID 71
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71
2024-03-15 10:23:16 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-71
2024-03-15 10:23:16 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:16 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:16 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:16 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:16 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:16 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 15
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15
2024-03-15 10:23:16 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:16 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=71 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f60cde6cd427e33da9a5486da999a6a with result from task 303
parsl.dataflow.dflow DEBUG: Task 303 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/3c4d61d0-3df5-49c7-9111-b3165ddf2845_isrForCrosstalkSources_995849_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/3c4d61d0-3df5-49c7-9111-b3165ddf2845_isrForCrosstalkSources_995849_4.stderr
parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3ee38e6a322fc79efda97470b71eca91 with result from task 298
parsl.dataflow.dflow DEBUG: Task 298 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c5d3e236-3933-448f-a59d-b9659c6d31a0_isrForCrosstalkSources_995993_34.stdout
parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c5d3e236-3933-448f-a59d-b9659c6d31a0_isrForCrosstalkSources_995993_34.stderr
parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry fb74b09029513f8b44dae10f10a37858 with result from task 296
parsl.dataflow.dflow DEBUG: Task 296 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/5147a3f4-fa08-411a-b496-ff4b3045dd62_isrForCrosstalkSources_995995_17.stdout
parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/5147a3f4-fa08-411a-b496-ff4b3045dd62_isrForCrosstalkSources_995995_17.stderr
parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f7189018200a799452744084d85cde05 with result from task 292
parsl.dataflow.dflow DEBUG: Task 292 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c638904c-019c-46eb-92f5-25e12292769e_isrForCrosstalkSources_995848_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/c638904c-019c-46eb-92f5-25e12292769e_isrForCrosstalkSources_995848_45.stderr
parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4e5c3ec21ad4dca90f03a4d59369139c with result from task 302
parsl.dataflow.dflow DEBUG: Task 302 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/57ebbd04-d184-4d28-80e3-970e4e2d352f_isrForCrosstalkSources_995850_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/57ebbd04-d184-4d28-80e3-970e4e2d352f_isrForCrosstalkSources_995850_47.stderr
parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a754c52dd6f18d8b0d2f975c00b8d501 with result from task 301
parsl.dataflow.dflow DEBUG: Task 301 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4e949c00-5249-4acd-8cbf-d22295dd77fb_isrForCrosstalkSources_995850_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/4e949c00-5249-4acd-8cbf-d22295dd77fb_isrForCrosstalkSources_995850_59.stderr
parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5853d4a86de65734cf5d345ce6ae663b with result from task 300
parsl.dataflow.dflow DEBUG: Task 300 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/7dbb20ba-386f-4587-9e01-8e61bb1d7a07_isrForCrosstalkSources_995849_27.stdout
parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/7dbb20ba-386f-4587-9e01-8e61bb1d7a07_isrForCrosstalkSources_995849_27.stderr
parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 54d96709a504b902dd5c7a94cb879222 with result from task 310
parsl.dataflow.dflow DEBUG: Task 310 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b0249783-8fde-47a5-9f6b-92205ba8b952_isrForCrosstalkSources_995849_7.stdout
parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/b0249783-8fde-47a5-9f6b-92205ba8b952_isrForCrosstalkSources_995849_7.stderr
parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1e171d933817b2296b67790be1063595 with result from task 299
parsl.dataflow.dflow DEBUG: Task 299 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/3bbf8147-4af9-43b5-952e-bbdd38a668a2_isrForCrosstalkSources_995849_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/3bbf8147-4af9-43b5-952e-bbdd38a668a2_isrForCrosstalkSources_995849_28.stderr
parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d85d9c63ed1481e448e95541b7d4d0e7 with result from task 304
parsl.dataflow.dflow DEBUG: Task 304 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/41bfc60a-a6fa-4a72-afe4-daf32a9e92f1_isrForCrosstalkSources_995848_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/41bfc60a-a6fa-4a72-afe4-daf32a9e92f1_isrForCrosstalkSources_995848_54.stderr
parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ce07bd99b7d93d0db95650b856248007 with result from task 313
parsl.dataflow.dflow DEBUG: Task 313 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/2613a58b-18f8-49e6-8aa1-d05fbfe30786_isrForCrosstalkSources_995993_45.stdout
parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/2613a58b-18f8-49e6-8aa1-d05fbfe30786_isrForCrosstalkSources_995993_45.stderr
parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c65fe74e4c7e1590b79a5b8c6b11b5f9 with result from task 305
parsl.dataflow.dflow DEBUG: Task 305 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/62027339-252a-4075-a070-7e474d65dad2_isrForCrosstalkSources_995994_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/62027339-252a-4075-a070-7e474d65dad2_isrForCrosstalkSources_995994_51.stderr
parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 040ff2d25e24f95e106cc2b784d04957 with result from task 297
parsl.dataflow.dflow DEBUG: Task 297 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4cf762b6-4daf-4b0b-9938-49cd3d1795dc_isrForCrosstalkSources_995994_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/4cf762b6-4daf-4b0b-9938-49cd3d1795dc_isrForCrosstalkSources_995994_44.stderr
2024-03-15 10:23:18 proc_lsst.multi:170 [INFO]  [multi] job_id 17064289
proc_lsst.multi INFO: [multi] job_id 17064289
2024-03-15 10:23:18 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-15 10:23:18 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:18 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064289
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064289
parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17064289
parsl.executors.status_handling INFO: Allocated block ID 72
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72
2024-03-15 10:23:18 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-72
2024-03-15 10:23:18 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:18 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:18 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:18 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:18 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:18 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 16
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16
2024-03-15 10:23:18 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:18 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=72 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 95ccdfb105c17dc460501f191c7866a1 with result from task 314
parsl.dataflow.dflow DEBUG: Task 314 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/02ef740b-02b4-496a-b043-abb7b3a2e37a_isrForCrosstalkSources_995849_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/02ef740b-02b4-496a-b043-abb7b3a2e37a_isrForCrosstalkSources_995849_29.stderr
parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 67f8e392a854d51c27ce90708c1e1ea6 with result from task 307
parsl.dataflow.dflow DEBUG: Task 307 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/94eaac7c-d870-49d5-a97c-66a83f40b7c9_isrForCrosstalkSources_995995_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/94eaac7c-d870-49d5-a97c-66a83f40b7c9_isrForCrosstalkSources_995995_46.stderr
parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 832f2b4e02e07cdd59779a982b682563 with result from task 312
parsl.dataflow.dflow DEBUG: Task 312 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9dc48048-4927-4438-a707-44ccb08aa4d3_isrForCrosstalkSources_995993_36.stdout
parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/9dc48048-4927-4438-a707-44ccb08aa4d3_isrForCrosstalkSources_995993_36.stderr
parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9b1ac921d9689279a5c61f090ba048dd with result from task 309
parsl.dataflow.dflow DEBUG: Task 309 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0c6eb885-24b7-43e2-99b3-f675afeb478d_isrForCrosstalkSources_995995_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/0c6eb885-24b7-43e2-99b3-f675afeb478d_isrForCrosstalkSources_995995_37.stderr
parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 25fabeff77742178b3157289fae2221f with result from task 306
parsl.dataflow.dflow DEBUG: Task 306 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3116a6b9-b118-4188-a3cf-831da4fdf247_isrForCrosstalkSources_995995_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/3116a6b9-b118-4188-a3cf-831da4fdf247_isrForCrosstalkSources_995995_57.stderr
parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 891ebbd7de75bfc7d2226a648ca8622d with result from task 311
parsl.dataflow.dflow DEBUG: Task 311 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/6bc22b26-35cf-44c6-9ae4-f029eb7f006a_isrForCrosstalkSources_995994_37.stdout
parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/6bc22b26-35cf-44c6-9ae4-f029eb7f006a_isrForCrosstalkSources_995994_37.stderr
parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry f583d7fafcc4d35cab81a4167e05b30f with result from task 329
parsl.dataflow.dflow DEBUG: Task 329 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5addadff-b512-4efb-941c-9191bb24b739_isrForCrosstalkSources_995848_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/5addadff-b512-4efb-941c-9191bb24b739_isrForCrosstalkSources_995848_26.stderr
parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b46e97d1e7d29970604ccb398b63b112 with result from task 326
parsl.dataflow.dflow DEBUG: Task 326 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/cec3d32d-6db6-4219-be57-54eeae93de11_isrForCrosstalkSources_995993_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/cec3d32d-6db6-4219-be57-54eeae93de11_isrForCrosstalkSources_995993_51.stderr
parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 3c113019fbb67bf9d9441c4e5e13c2a7 with result from task 322
parsl.dataflow.dflow DEBUG: Task 322 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d6e53171-6a06-43b8-bdd4-11c008648106_isrForCrosstalkSources_995850_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d6e53171-6a06-43b8-bdd4-11c008648106_isrForCrosstalkSources_995850_56.stderr
parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 22f28f828cf3ba103527c8d25f3c86a0 with result from task 308
parsl.dataflow.dflow DEBUG: Task 308 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/e57c06de-1c81-4e96-867e-4523aee7a1af_isrForCrosstalkSources_995850_38.stdout
parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/e57c06de-1c81-4e96-867e-4523aee7a1af_isrForCrosstalkSources_995850_38.stderr
parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 32c01df6d96b56ee66f65b62c98ec21f with result from task 324
parsl.dataflow.dflow DEBUG: Task 324 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f27755ef-5817-41c3-a037-86ab1b6a70b0_isrForCrosstalkSources_995995_24.stdout
parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f27755ef-5817-41c3-a037-86ab1b6a70b0_isrForCrosstalkSources_995995_24.stderr
parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa759d015d1f16f1695dbd2c2c572a1 with result from task 321
parsl.dataflow.dflow DEBUG: Task 321 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/d42ecdc1-4dc4-48b7-991e-73594ee3e90e_isrForCrosstalkSources_995994_5.stdout
parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/d42ecdc1-4dc4-48b7-991e-73594ee3e90e_isrForCrosstalkSources_995994_5.stderr
parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f002a5efb5fdccd7ebd9c5018e256a4 with result from task 315
parsl.dataflow.dflow DEBUG: Task 315 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/edc2277e-fcc7-4d47-9895-5f7ddf8a8459_isrForCrosstalkSources_995994_16.stdout
parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/edc2277e-fcc7-4d47-9895-5f7ddf8a8459_isrForCrosstalkSources_995994_16.stderr
parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8607f9e6c8ef17644f0e264bca62beed with result from task 319
parsl.dataflow.dflow DEBUG: Task 319 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/29deb951-1297-4da6-b59b-ff05958379ac_isrForCrosstalkSources_995995_32.stdout
parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/29deb951-1297-4da6-b59b-ff05958379ac_isrForCrosstalkSources_995995_32.stderr
2024-03-15 10:23:20 proc_lsst.multi:170 [INFO]  [multi] job_id 17064290
proc_lsst.multi INFO: [multi] job_id 17064290
2024-03-15 10:23:20 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-15 10:23:20 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:20 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064290
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064290
parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17064290
parsl.executors.status_handling INFO: Allocated block ID 73
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73
2024-03-15 10:23:20 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-73
2024-03-15 10:23:20 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:20 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:20 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:20 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:20 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:20 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 17
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17
2024-03-15 10:23:20 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:20 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=73 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry d3a202ab9c639d1259cf553a96e86139 with result from task 318
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow DEBUG: Task 318 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fd870f20-1620-4f0a-ac0a-6a48081d7e51_isrForCrosstalkSources_995995_43.stdout
parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/fd870f20-1620-4f0a-ac0a-6a48081d7e51_isrForCrosstalkSources_995995_43.stderr
parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 63636ca60e2669f878e1adf0a35431cc with result from task 331
parsl.dataflow.dflow DEBUG: Task 331 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/52412165-f720-425a-b844-efaf60221448_isrForCrosstalkSources_995849_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/52412165-f720-425a-b844-efaf60221448_isrForCrosstalkSources_995849_56.stderr
parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 48cc62c4538d05ae53de94c9d356e014 with result from task 325
parsl.dataflow.dflow DEBUG: Task 325 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/7e2a64f3-ab88-4fb8-977d-7b232c0cc127_isrForCrosstalkSources_995995_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/7e2a64f3-ab88-4fb8-977d-7b232c0cc127_isrForCrosstalkSources_995995_55.stderr
parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 648ef8e0e80bb222c228a3df6bc14f10 with result from task 320
parsl.dataflow.dflow DEBUG: Task 320 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/e43e1272-f8f3-4d1a-a32f-73c0a50e75e6_isrForCrosstalkSources_995850_11.stdout
parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/e43e1272-f8f3-4d1a-a32f-73c0a50e75e6_isrForCrosstalkSources_995850_11.stderr
parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 10061cc1992ad104628a6bd12b4d5f51 with result from task 316
parsl.dataflow.dflow DEBUG: Task 316 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/dcd48855-eb7e-4d49-ba3a-1306e18b1c4c_isrForCrosstalkSources_995849_26.stdout
parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/dcd48855-eb7e-4d49-ba3a-1306e18b1c4c_isrForCrosstalkSources_995849_26.stderr
parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0943c9ed4453977060a1d3e4135da330 with result from task 323
parsl.dataflow.dflow DEBUG: Task 323 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/c89b2711-279f-4622-bd35-547c7357cdbf_isrForCrosstalkSources_995994_3.stdout
parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/c89b2711-279f-4622-bd35-547c7357cdbf_isrForCrosstalkSources_995994_3.stderr
parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5868a0079cb607dfc37a22912e647233 with result from task 317
parsl.dataflow.dflow DEBUG: Task 317 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e978b32e-0464-4e63-ba09-b92a7847b38b_isrForCrosstalkSources_995993_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/e978b32e-0464-4e63-ba09-b92a7847b38b_isrForCrosstalkSources_995993_57.stderr
parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry aa85dc1559556c2add5dcd9bd710a21e with result from task 328
parsl.dataflow.dflow DEBUG: Task 328 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/df7e57c6-444e-4a06-b083-189a7fccc6ab_isrForCrosstalkSources_995995_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/df7e57c6-444e-4a06-b083-189a7fccc6ab_isrForCrosstalkSources_995995_44.stderr
parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 875817aafc8217778da607b086a2cb26 with result from task 327
parsl.dataflow.dflow DEBUG: Task 327 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/72ad8e77-c5c2-4930-a249-b061512d8d73_isrForCrosstalkSources_995850_48.stdout
parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/72ad8e77-c5c2-4930-a249-b061512d8d73_isrForCrosstalkSources_995850_48.stderr
parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a6c38baa7be1182af6688e4fa2f20956 with result from task 332
parsl.dataflow.dflow DEBUG: Task 332 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/534df77b-f648-4f23-a6a3-351970010611_isrForCrosstalkSources_995993_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/534df77b-f648-4f23-a6a3-351970010611_isrForCrosstalkSources_995993_46.stderr
2024-03-15 10:23:22 proc_lsst.multi:170 [INFO]  [multi] job_id 17064291
proc_lsst.multi INFO: [multi] job_id 17064291
2024-03-15 10:23:22 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-15 10:23:22 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:22 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064291
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064291
parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17064291
parsl.executors.status_handling INFO: Allocated block ID 74
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74
2024-03-15 10:23:22 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-74
2024-03-15 10:23:22 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:22 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:22 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:22 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:22 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:22 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 18
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18
2024-03-15 10:23:22 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:22 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=74 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 8317fbbf46eae0562cca9f0a3ac666a8 with result from task 334
parsl.dataflow.dflow DEBUG: Task 334 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/708e2038-9c69-49d6-abd1-5389a21c6881_isrForCrosstalkSources_995849_2.stdout
parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/708e2038-9c69-49d6-abd1-5389a21c6881_isrForCrosstalkSources_995849_2.stderr
parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry a99404eeaa69e4078e1ac41f5711a6e7 with result from task 338
parsl.dataflow.dflow DEBUG: Task 338 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/17c65322-78ef-441d-b5b4-67687afdde72_isrForCrosstalkSources_995995_39.stdout
parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/17c65322-78ef-441d-b5b4-67687afdde72_isrForCrosstalkSources_995995_39.stderr
parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dd502c5899193c4acd54794648d19d0b with result from task 336
parsl.dataflow.dflow DEBUG: Task 336 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/2c69f874-d374-44e5-8b64-8725b6fc2022_isrForCrosstalkSources_995993_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/2c69f874-d374-44e5-8b64-8725b6fc2022_isrForCrosstalkSources_995993_19.stderr
parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1139b61db5a1cf62d3b25b8d9611f444 with result from task 333
parsl.dataflow.dflow DEBUG: Task 333 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0a212b10-08cb-47f5-91e5-6d3d180b379d_isrForCrosstalkSources_995848_59.stdout
parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/0a212b10-08cb-47f5-91e5-6d3d180b379d_isrForCrosstalkSources_995848_59.stderr
parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eb70101b2f630dc405ae551667182d7f with result from task 335
parsl.dataflow.dflow DEBUG: Task 335 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/163c3f9a-9e83-4b7a-8289-173611d2737b_isrForCrosstalkSources_995995_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/163c3f9a-9e83-4b7a-8289-173611d2737b_isrForCrosstalkSources_995995_31.stderr
parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry c32318e46a06b6907c999ae1b133adb6 with result from task 343
parsl.dataflow.dflow DEBUG: Task 343 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/76e1b35c-3c83-4ae6-b6e4-05abc468082a_isrForCrosstalkSources_995849_15.stdout
parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/76e1b35c-3c83-4ae6-b6e4-05abc468082a_isrForCrosstalkSources_995849_15.stderr
parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 337c4d3b4ea1246793c366b79d4943cb with result from task 348
parsl.dataflow.dflow DEBUG: Task 348 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/929beb7e-94c9-4357-8e46-e6a0bbc9ccab_isrForCrosstalkSources_995849_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/929beb7e-94c9-4357-8e46-e6a0bbc9ccab_isrForCrosstalkSources_995849_13.stderr
parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4f961f327d56afb8dd8b1202b84562fc with result from task 340
parsl.dataflow.dflow DEBUG: Task 340 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ef4c03e1-53cf-4d3d-afb3-985dc8b44551_isrForCrosstalkSources_995849_46.stdout
parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/ef4c03e1-53cf-4d3d-afb3-985dc8b44551_isrForCrosstalkSources_995849_46.stderr
2024-03-15 10:23:23 proc_lsst.multi:170 [INFO]  [multi] job_id 17064292
proc_lsst.multi INFO: [multi] job_id 17064292
2024-03-15 10:23:23 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-15 10:23:23 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:23 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064292
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064292
parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17064292
parsl.executors.status_handling INFO: Allocated block ID 75
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75
2024-03-15 10:23:23 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-75
2024-03-15 10:23:23 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:23 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:23 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:23 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:23 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:23 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 19
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19
2024-03-15 10:23:23 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:23 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=75 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 116ae818290f83f0d00ea17263247f0d with result from task 337
parsl.dataflow.dflow DEBUG: Task 337 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a926d484-ddab-4a6a-a4d1-b9d4a10993de_isrForCrosstalkSources_995995_42.stdout
parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/a926d484-ddab-4a6a-a4d1-b9d4a10993de_isrForCrosstalkSources_995995_42.stderr
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry acb8ffe06ba694d6f815c8e0efa3338f with result from task 341
parsl.dataflow.dflow DEBUG: Task 341 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/0561368a-2e23-42c6-b65c-43a188fda943_isrForCrosstalkSources_995993_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/0561368a-2e23-42c6-b65c-43a188fda943_isrForCrosstalkSources_995993_31.stderr
parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 58599a15e7ce7eeaf3a7b1c9f960be48 with result from task 342
parsl.dataflow.dflow DEBUG: Task 342 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/afa5be02-ba30-4f50-8dc1-3cfd774fdccc_isrForCrosstalkSources_995850_25.stdout
parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/afa5be02-ba30-4f50-8dc1-3cfd774fdccc_isrForCrosstalkSources_995850_25.stderr
parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 00c28e53725bceb1eb3c9cb8b5e55a83 with result from task 344
parsl.dataflow.dflow DEBUG: Task 344 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/b261c2c6-9d28-4051-a8c3-65132590acc7_isrForCrosstalkSources_995848_58.stdout
parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/b261c2c6-9d28-4051-a8c3-65132590acc7_isrForCrosstalkSources_995848_58.stderr
parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 9628c80ffebe96017ebac93d28c77748 with result from task 345
parsl.dataflow.dflow DEBUG: Task 345 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/17d54452-91bf-47a4-83be-138176d76257_isrForCrosstalkSources_995848_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/17d54452-91bf-47a4-83be-138176d76257_isrForCrosstalkSources_995848_50.stderr
parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5642387bc008969486decb20db9fec8b with result from task 349
parsl.dataflow.dflow DEBUG: Task 349 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2c5045f3-b4ab-4fa7-97e0-82538b05ea58_isrForCrosstalkSources_995995_61.stdout
parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2c5045f3-b4ab-4fa7-97e0-82538b05ea58_isrForCrosstalkSources_995995_61.stderr
parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry eb4458e1ac572a5364c0779eb2383e35 with result from task 339
parsl.dataflow.dflow DEBUG: Task 339 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/5798c9bb-af40-443d-a399-d3771f7e9576_isrForCrosstalkSources_995849_54.stdout
parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/5798c9bb-af40-443d-a399-d3771f7e9576_isrForCrosstalkSources_995849_54.stderr
parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 19b98ddd1de00fe44686ccdf4db17097 with result from task 346
parsl.dataflow.dflow DEBUG: Task 346 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/eab0ebd9-1872-4cec-8925-a305a4bf5fe7_isrForCrosstalkSources_995995_50.stdout
parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/eab0ebd9-1872-4cec-8925-a305a4bf5fe7_isrForCrosstalkSources_995995_50.stderr
parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7fba3d05e9ae65c1105c8b55aeec6a61 with result from task 350
parsl.dataflow.dflow DEBUG: Task 350 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/7aeb7c8a-7a87-4900-b492-b24a94791f7e_isrForCrosstalkSources_995848_56.stdout
parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/7aeb7c8a-7a87-4900-b492-b24a94791f7e_isrForCrosstalkSources_995848_56.stderr
parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5b7096f1f0955c93f4352137e3e37f8f with result from task 352
parsl.dataflow.dflow DEBUG: Task 352 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f5959628-b34c-4051-9b88-b8a7007594dd_isrForCrosstalkSources_995995_13.stdout
parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/f5959628-b34c-4051-9b88-b8a7007594dd_isrForCrosstalkSources_995995_13.stderr
2024-03-15 10:23:25 proc_lsst.multi:170 [INFO]  [multi] job_id 17064294
proc_lsst.multi INFO: [multi] job_id 17064294
2024-03-15 10:23:25 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:25 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:25 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064294
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064294
parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17064294
parsl.executors.status_handling INFO: Allocated block ID 76
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76
2024-03-15 10:23:25 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-76
2024-03-15 10:23:25 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:25 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:25 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:25 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:25 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:25 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:25 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:25 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=76 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4976480e1440c2dd5a7b0a431928a094 with result from task 330
parsl.dataflow.dflow DEBUG: Task 330 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d676e161-eb7e-42dc-a592-068d05a85013_isrForCrosstalkSources_995850_41.stdout
parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d676e161-eb7e-42dc-a592-068d05a85013_isrForCrosstalkSources_995850_41.stderr
parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 262a77dcf016c8cb40796a0c5732318e with result from task 351
parsl.dataflow.dflow DEBUG: Task 351 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/bdfe1e4b-73f3-44c8-b55c-7c02a080592f_isrForCrosstalkSources_995993_49.stdout
parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/bdfe1e4b-73f3-44c8-b55c-7c02a080592f_isrForCrosstalkSources_995993_49.stderr
parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 692ddacd912e8414c78aa3241fd734f6 with result from task 362
parsl.dataflow.dflow DEBUG: Task 362 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/16fe9034-5770-4aed-8c9f-cee3d987705e_isrForCrosstalkSources_995993_10.stdout
parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/16fe9034-5770-4aed-8c9f-cee3d987705e_isrForCrosstalkSources_995993_10.stderr
2024-03-15 10:23:26 proc_lsst.multi:170 [INFO]  [multi] job_id 17064296
proc_lsst.multi INFO: [multi] job_id 17064296
2024-03-15 10:23:26 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-15 10:23:26 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:26 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064296
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064296
parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17064296
parsl.executors.status_handling INFO: Allocated block ID 77
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77
2024-03-15 10:23:26 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-77
2024-03-15 10:23:26 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:26 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:26 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:26 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:26 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:26 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 21
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21
2024-03-15 10:23:26 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:26 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=77 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 36b34b88aed76b6cfef9ccef39c2e417 with result from task 358
parsl.dataflow.dflow DEBUG: Task 358 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f783b177-d75a-4532-a35b-52c09a87dde7_isrForCrosstalkSources_995994_4.stdout
parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/f783b177-d75a-4532-a35b-52c09a87dde7_isrForCrosstalkSources_995994_4.stderr
parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry dcfea24c52fd2b71e78fd323d512f114 with result from task 347
parsl.dataflow.dflow DEBUG: Task 347 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/7a2027af-7c21-45da-bc96-cfbc64d35092_isrForCrosstalkSources_995995_60.stdout
parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/7a2027af-7c21-45da-bc96-cfbc64d35092_isrForCrosstalkSources_995995_60.stderr
parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0d0640307096e103a7ce40def50f16a8 with result from task 353
parsl.dataflow.dflow DEBUG: Task 353 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bf311e54-3cdd-418e-9444-f7b1ca6a3b04_isrForCrosstalkSources_995850_19.stdout
parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/bf311e54-3cdd-418e-9444-f7b1ca6a3b04_isrForCrosstalkSources_995850_19.stderr
parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 0ffd28f7d45ab26d1df954532adb671b with result from task 354
parsl.dataflow.dflow DEBUG: Task 354 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/52030bea-9c7c-4cf5-8755-dd8031b39fe8_isrForCrosstalkSources_995850_51.stdout
parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/52030bea-9c7c-4cf5-8755-dd8031b39fe8_isrForCrosstalkSources_995850_51.stderr
parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5d8b6b75f6194c498e28f24981012e7 with result from task 367
parsl.dataflow.dflow DEBUG: Task 367 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/83c14bfd-3dae-42df-93b5-358a7bf6b0cd_isrForCrosstalkSources_995993_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/83c14bfd-3dae-42df-93b5-358a7bf6b0cd_isrForCrosstalkSources_995993_47.stderr
2024-03-15 10:23:27 proc_lsst.multi:170 [INFO]  [multi] job_id 17064297
proc_lsst.multi INFO: [multi] job_id 17064297
2024-03-15 10:23:27 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-15 10:23:27 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:27 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064297
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064297
parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17064297
parsl.executors.status_handling INFO: Allocated block ID 78
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78
2024-03-15 10:23:27 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-78
2024-03-15 10:23:27 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:27 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:27 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:27 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:27 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:27 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 22
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22
2024-03-15 10:23:27 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:27 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=78 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry b5e6c7fd9148e772126374395fb36294 with result from task 360
parsl.dataflow.dflow DEBUG: Task 360 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d6d73757-28c8-4d09-ac3d-23ec005f1748_isrForCrosstalkSources_995850_1.stdout
parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/d6d73757-28c8-4d09-ac3d-23ec005f1748_isrForCrosstalkSources_995850_1.stderr
2024-03-15 10:23:28 proc_lsst.multi:170 [INFO]  [multi] job_id 17064298
proc_lsst.multi INFO: [multi] job_id 17064298
2024-03-15 10:23:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-15 10:23:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:28 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064298
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064298
parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17064298
parsl.executors.status_handling INFO: Allocated block ID 79
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79
2024-03-15 10:23:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-79
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 23
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=79 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 341ced2364eb47c3a7280221c1533ded with result from task 364
parsl.dataflow.dflow DEBUG: Task 364 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/d3014836-44af-462b-ab5f-24b653839644_isrForCrosstalkSources_995993_20.stdout
parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/d3014836-44af-462b-ab5f-24b653839644_isrForCrosstalkSources_995993_20.stderr
parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 90eb9641c8a33b61d146bc0073519af2 with result from task 359
parsl.dataflow.dflow DEBUG: Task 359 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5e7093c3-7995-447b-bc1e-c3f4726413ca_isrForCrosstalkSources_995994_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/5e7093c3-7995-447b-bc1e-c3f4726413ca_isrForCrosstalkSources_995994_28.stderr
parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 771f3fd9628284c37a09ce57e057e570 with result from task 365
parsl.dataflow.dflow DEBUG: Task 365 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/19630532-dd96-4a57-bcba-702087780d18_isrForCrosstalkSources_995850_52.stdout
parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995850/19630532-dd96-4a57-bcba-702087780d18_isrForCrosstalkSources_995850_52.stderr
parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 5675f3508eb7063150d561355ad39528 with result from task 355
parsl.dataflow.dflow DEBUG: Task 355 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2cdab690-56d5-45f7-ae34-eeec1b6c27e4_isrForCrosstalkSources_995995_12.stdout
parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995995/2cdab690-56d5-45f7-ae34-eeec1b6c27e4_isrForCrosstalkSources_995995_12.stderr
parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 1558165efcb9512377f6e7f524de9259 with result from task 357
parsl.dataflow.dflow DEBUG: Task 357 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/74a0b775-6702-477b-8ad5-5f1512acfbba_isrForCrosstalkSources_995994_47.stdout
parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/74a0b775-6702-477b-8ad5-5f1512acfbba_isrForCrosstalkSources_995994_47.stderr
parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 7b14e8dd1c940575c95da7070eeab1eb with result from task 356
parsl.dataflow.dflow DEBUG: Task 356 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/45e78ff1-4699-498c-91b5-1a006e8ff33a_isrForCrosstalkSources_995849_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995849/45e78ff1-4699-498c-91b5-1a006e8ff33a_isrForCrosstalkSources_995849_55.stderr
parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 05fc0694500e3d19ec571b521d9c908b with result from task 369
parsl.dataflow.dflow DEBUG: Task 369 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/1fbc3e81-4743-4383-95c5-673d4e8ddbc8_isrForCrosstalkSources_995848_31.stdout
parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/1fbc3e81-4743-4383-95c5-673d4e8ddbc8_isrForCrosstalkSources_995848_31.stderr
2024-03-15 10:23:28 proc_lsst.multi:170 [INFO]  [multi] job_id 17064299
proc_lsst.multi INFO: [multi] job_id 17064299
2024-03-15 10:23:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-15 10:23:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:28 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064299
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064299
parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17064299
parsl.executors.status_handling INFO: Allocated block ID 80
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80
2024-03-15 10:23:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-80
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 24
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=80 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry ba92faf33bc0622d6ae4d92c55616c8a with result from task 371
parsl.dataflow.dflow DEBUG: Task 371 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c105e832-2cbe-4bb3-8c20-8df2a3e32231_isrForCrosstalkSources_995993_28.stdout
parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/c105e832-2cbe-4bb3-8c20-8df2a3e32231_isrForCrosstalkSources_995993_28.stderr
parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 136d802c7a4904f54a3c403a70ed7d9a with result from task 366
parsl.dataflow.dflow DEBUG: Task 366 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a77487d0-264c-4673-b941-d3e92b38cc21_isrForCrosstalkSources_995994_57.stdout
parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/a77487d0-264c-4673-b941-d3e92b38cc21_isrForCrosstalkSources_995994_57.stderr
parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 4a23e139ce87e645522dce3d20bbadb2 with result from task 361
parsl.dataflow.dflow DEBUG: Task 361 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ba66456c-d6c5-469c-a6af-1c68020afab9_isrForCrosstalkSources_995848_55.stdout
parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995848/ba66456c-d6c5-469c-a6af-1c68020afab9_isrForCrosstalkSources_995848_55.stderr
parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry 55537c2700488cc2ec89424ebd7baa04 with result from task 363
parsl.dataflow.dflow DEBUG: Task 363 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/eb5d3ba5-aa80-49e2-ad28-c45dfba37884_isrForCrosstalkSources_995994_62.stdout
parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995994/eb5d3ba5-aa80-49e2-ad28-c45dfba37884_isrForCrosstalkSources_995994_62.stderr
2024-03-15 10:23:28 proc_lsst.multi:170 [INFO]  [multi] job_id 17064300
proc_lsst.multi INFO: [multi] job_id 17064300
2024-03-15 10:23:28 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-15 10:23:28 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:28 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064300
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064300
parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17064300
parsl.executors.status_handling INFO: Allocated block ID 81
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81
2024-03-15 10:23:28 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-81
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:28 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:28 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 25
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25
2024-03-15 10:23:28 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:28 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=81 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry e307b4487254597a110595233e95cbb9 with result from task 370
parsl.dataflow.dflow DEBUG: Task 370 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/47bc7457-77ba-4840-a223-59bd958245d5_isrForCrosstalkSources_995993_29.stdout
parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/47bc7457-77ba-4840-a223-59bd958245d5_isrForCrosstalkSources_995993_29.stderr
parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done)
parsl.dataflow.memoization DEBUG: Storing app cache entry da5fd4bd7c04c39f9578312aabfe5065 with result from task 368
parsl.dataflow.dflow INFO: DFK cleanup initiated
parsl.dataflow.dflow INFO: Summary of tasks in DFK:
parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0
parsl.dataflow.dflow INFO: Tasks in state States.pending: 0
parsl.dataflow.dflow INFO: Tasks in state States.running: 0
parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 372
parsl.dataflow.dflow INFO: Tasks in state States.failed: 0
parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0
parsl.dataflow.dflow INFO: Tasks in state States.launched: 0
parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0
parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0
parsl.dataflow.dflow INFO: Tasks in state States.joining: 0
parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0
parsl.dataflow.dflow INFO: End of summary
parsl.dataflow.dflow DEBUG: Task 368 checkpointed
parsl.dataflow.dflow INFO: Done checkpointing 1 tasks
parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6452b5cc-9110-40d0-b0b9-a4ffd3a3694c_isrForCrosstalkSources_995993_44.stdout
parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/logs/isrForCrosstalkSources/995993/6452b5cc-9110-40d0-b0b9-a4ffd3a3694c_isrForCrosstalkSources_995993_44.stderr
parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass.
parsl.dataflow.dflow INFO: Closing job status poller
2024-03-15 10:23:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17064301
proc_lsst.multi INFO: [multi] job_id 17064301
2024-03-15 10:23:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-15 10:23:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064301
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064301
parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17064301
parsl.executors.status_handling INFO: Allocated block ID 82
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82
2024-03-15 10:23:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-82
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 26
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=82 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17064302
proc_lsst.multi INFO: [multi] job_id 17064302
2024-03-15 10:23:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-15 10:23:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064302
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064302
parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17064302
parsl.executors.status_handling INFO: Allocated block ID 83
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83
2024-03-15 10:23:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-83
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 27
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=83 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17064303
proc_lsst.multi INFO: [multi] job_id 17064303
2024-03-15 10:23:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-15 10:23:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064303
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064303
parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17064303
parsl.executors.status_handling INFO: Allocated block ID 84
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84
2024-03-15 10:23:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-84
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 28
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=84 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17064304
proc_lsst.multi INFO: [multi] job_id 17064304
2024-03-15 10:23:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-15 10:23:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064304
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064304
parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17064304
parsl.executors.status_handling INFO: Allocated block ID 85
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85
2024-03-15 10:23:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-85
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 29
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=85 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17064305
proc_lsst.multi INFO: [multi] job_id 17064305
2024-03-15 10:23:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064305
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064305
parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17064305
parsl.executors.status_handling INFO: Allocated block ID 86
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86
2024-03-15 10:23:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-86
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=86 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:29 proc_lsst.multi:170 [INFO]  [multi] job_id 17064306
proc_lsst.multi INFO: [multi] job_id 17064306
2024-03-15 10:23:29 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-15 10:23:29 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064306
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064306
parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17064306
parsl.executors.status_handling INFO: Allocated block ID 87
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87
2024-03-15 10:23:29 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-87
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:29 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:29 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 31
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31
2024-03-15 10:23:29 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:29 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=87 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17064307
proc_lsst.multi INFO: [multi] job_id 17064307
2024-03-15 10:23:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-15 10:23:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064307
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064307
parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17064307
parsl.executors.status_handling INFO: Allocated block ID 88
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88
2024-03-15 10:23:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-88
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 32
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=88 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17064308
proc_lsst.multi INFO: [multi] job_id 17064308
2024-03-15 10:23:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-15 10:23:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064308
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064308
parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17064308
parsl.executors.status_handling INFO: Allocated block ID 89
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89
2024-03-15 10:23:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-89
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 33
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=89 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17064309
proc_lsst.multi INFO: [multi] job_id 17064309
2024-03-15 10:23:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-15 10:23:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064309
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064309
parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17064309
parsl.executors.status_handling INFO: Allocated block ID 90
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90
2024-03-15 10:23:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-90
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 34
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=90 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17064310
proc_lsst.multi INFO: [multi] job_id 17064310
2024-03-15 10:23:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-15 10:23:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064310
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064310
parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17064310
parsl.executors.status_handling INFO: Allocated block ID 91
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91
2024-03-15 10:23:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-91
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 35
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=91 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17064311
proc_lsst.multi INFO: [multi] job_id 17064311
2024-03-15 10:23:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-15 10:23:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064311
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064311
parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17064311
parsl.executors.status_handling INFO: Allocated block ID 92
parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92
2024-03-15 10:23:30 proc_lsst.multi:153 [INFO]  [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
proc_lsst.multi INFO: [multi] got submit process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn 1 parsl.multi.block-92
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
proc_lsst.multi INFO: [multi] local process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn --max_workers 2
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 1
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 1
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] astro process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 30
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 30
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 5
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 5
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 20
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 20
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20
2024-03-15 10:23:30 proc_lsst.multi:162 [INFO]  [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn
2024-03-15 10:23:30 proc_lsst.multi:163 [INFO]  [multi] len(self.providers[provider].resources) 36
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36
2024-03-15 10:23:30 proc_lsst.multi:164 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:166 [INFO]  [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
proc_lsst.multi INFO: [multi] submitting process_worker_pool.py   -a 127.0.0.1,n3009,198.48.92.26,10.64.65.9,169.254.95.120,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54274 --result_port=54751 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1h/science#step0/20240315T102059Z/runinfo/000/multi --block_id=92 --hb_period=30  --hb_threshold=120 --cpu-affinity none --available-accelerators  --start-method spawn to checkpoint
parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes
parsl.providers.slurm.slurm DEBUG: Writing submit script
parsl.providers.slurm.slurm DEBUG: moving files
2024-03-15 10:23:30 proc_lsst.multi:170 [INFO]  [multi] job_id 17064312
proc_lsst.multi INFO: [multi] job_id 17064312
2024-03-15 10:23:30 proc_lsst.multi:171 [INFO]  [multi] len(self.providers[provider].resources) 37
proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37
2024-03-15 10:23:30 proc_lsst.multi:172 [INFO]  [multi] self.providers[provider].max_blocks 256
proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256
2024-03-15 10:23:30 proc_lsst.multi:178 [INFO]  [multi] provider checkpoint accepted submit and returned 17064312
proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17064312
parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17064312
parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22647872635152
parsl.dataflow.dflow INFO: Terminated job status poller
parsl.dataflow.dflow INFO: Scaling in and shutting down executors
parsl.dataflow.dflow INFO: Scaling in executor multi
parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[]
parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 56 blocks
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e9cd10a58abd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e9cd10a58abd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d2bd6a0bafa
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d2bd6a0bafa
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fff2d658b60e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fff2d658b60e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e93a6daa664f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e93a6daa664f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f912902a305
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f912902a305
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62d2b7632b44
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62d2b7632b44
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9386cee3dc2d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9386cee3dc2d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c98431c1259
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c98431c1259
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 95168a8e6b22
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 95168a8e6b22
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 61fb251927d2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 61fb251927d2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ad0011a5790
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ad0011a5790
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: db8d39f2e1fb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: db8d39f2e1fb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b24794026676
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b24794026676
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67b01dc89433
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67b01dc89433
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67da20987c33
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67da20987c33
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0f96e3b20014
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0f96e3b20014
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 83b9c9d1b21f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 83b9c9d1b21f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 10433362ef0f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 10433362ef0f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9248d7baab0e
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9248d7baab0e
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c2ab1e20aff3
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c2ab1e20aff3
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ef5fcbfb68e9
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ef5fcbfb68e9
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 51949a685ef0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 51949a685ef0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 899c18580cd8
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 899c18580cd8
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 39572116a136
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 39572116a136
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fd552cf7d87b
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fd552cf7d87b
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0390f1aad3af
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0390f1aad3af
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: af8f03c81de0
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: af8f03c81de0
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7124491ba31c
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7124491ba31c
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df6ec7e99ebd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df6ec7e99ebd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b475e3a5f5eb
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b475e3a5f5eb
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7ec0177222ff
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7ec0177222ff
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cc879493eadc
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cc879493eadc
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1b89932bce98
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1b89932bce98
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 60606abb97d6
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 60606abb97d6
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3fadac9a5c21
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3fadac9a5c21
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6ff634df84b7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6ff634df84b7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8c5075584431
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8c5075584431
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e93916ab82ce
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e93916ab82ce
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1f62dcaaad58
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1f62dcaaad58
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 50ddb5886869
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 50ddb5886869
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 909fad5935a7
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 909fad5935a7
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b73ade900118
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b73ade900118
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9fea1a883153
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9fea1a883153
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dcd5854931f2
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dcd5854931f2
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 68075e884e50
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 68075e884e50
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: beeaace374ba
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: beeaace374ba
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 968d0829e1cd
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 968d0829e1cd
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df1fd8e7c7b4
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df1fd8e7c7b4
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5aa6022d8d32
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5aa6022d8d32
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f81d3ee8508
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f81d3ee8508
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 326edc9a0d81
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 326edc9a0d81
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 177520f64828
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 177520f64828
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d00c2971d33d
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d00c2971d33d
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 63c1724c5e8f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 63c1724c5e8f
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d5f6c3c2861
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d5f6c3c2861
parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5bf1bd9ffe5f
parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5bf1bd9ffe5f
2024-03-15 10:23:31 proc_lsst.multi:146 [INFO]  found job 17064169 in provider astro
proc_lsst.multi INFO: found job 17064169 in provider astro
2024-03-15 10:23:31 proc_lsst.multi:201 [INFO]  cancelling 17064169 on provider astro
proc_lsst.multi INFO: cancelling 17064169 on provider astro
2024-03-15 10:23:31 proc_lsst.multi:146 [INFO]  found job 17064173 in provider astro
proc_lsst.multi INFO: found job 17064173 in provider astro
2024-03-15 10:23:31 proc_lsst.multi:201 [INFO]  cancelling 17064173 on provider astro
proc_lsst.multi INFO: cancelling 17064173 on provider astro
2024-03-15 10:23:31 proc_lsst.multi:146 [INFO]  found job 17064180 in provider astro
proc_lsst.multi INFO: found job 17064180 in provider astro
2024-03-15 10:23:31 proc_lsst.multi:201 [INFO]  cancelling 17064180 on provider astro
proc_lsst.multi INFO: cancelling 17064180 on provider astro
2024-03-15 10:23:31 proc_lsst.multi:146 [INFO]  found job 17064222 in provider escience_a40
proc_lsst.multi INFO: found job 17064222 in provider escience_a40
2024-03-15 10:23:31 proc_lsst.multi:201 [INFO]  cancelling 17064222 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064222 on provider escience_a40
2024-03-15 10:23:31 proc_lsst.multi:146 [INFO]  found job 17064171 in provider astro
proc_lsst.multi INFO: found job 17064171 in provider astro
2024-03-15 10:23:31 proc_lsst.multi:201 [INFO]  cancelling 17064171 on provider astro
proc_lsst.multi INFO: cancelling 17064171 on provider astro
2024-03-15 10:23:31 proc_lsst.multi:146 [INFO]  found job 17064243 in provider escience_a40
proc_lsst.multi INFO: found job 17064243 in provider escience_a40
2024-03-15 10:23:31 proc_lsst.multi:201 [INFO]  cancelling 17064243 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064243 on provider escience_a40
2024-03-15 10:23:32 proc_lsst.multi:146 [INFO]  found job 17064183 in provider astro
proc_lsst.multi INFO: found job 17064183 in provider astro
2024-03-15 10:23:32 proc_lsst.multi:201 [INFO]  cancelling 17064183 on provider astro
proc_lsst.multi INFO: cancelling 17064183 on provider astro
2024-03-15 10:23:32 proc_lsst.multi:146 [INFO]  found job 17064223 in provider escience_a40
proc_lsst.multi INFO: found job 17064223 in provider escience_a40
2024-03-15 10:23:32 proc_lsst.multi:201 [INFO]  cancelling 17064223 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064223 on provider escience_a40
2024-03-15 10:23:32 proc_lsst.multi:146 [INFO]  found job 17064215 in provider escience_a40
proc_lsst.multi INFO: found job 17064215 in provider escience_a40
2024-03-15 10:23:32 proc_lsst.multi:201 [INFO]  cancelling 17064215 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064215 on provider escience_a40
2024-03-15 10:23:32 proc_lsst.multi:146 [INFO]  found job 17064208 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064208 in provider escience_rtx6k
2024-03-15 10:23:32 proc_lsst.multi:201 [INFO]  cancelling 17064208 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064208 on provider escience_rtx6k
2024-03-15 10:23:32 proc_lsst.multi:146 [INFO]  found job 17064233 in provider escience_a40
proc_lsst.multi INFO: found job 17064233 in provider escience_a40
2024-03-15 10:23:32 proc_lsst.multi:201 [INFO]  cancelling 17064233 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064233 on provider escience_a40
2024-03-15 10:23:32 proc_lsst.multi:146 [INFO]  found job 17064221 in provider escience_a40
proc_lsst.multi INFO: found job 17064221 in provider escience_a40
2024-03-15 10:23:32 proc_lsst.multi:201 [INFO]  cancelling 17064221 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064221 on provider escience_a40
2024-03-15 10:23:32 proc_lsst.multi:146 [INFO]  found job 6561 in provider local
proc_lsst.multi INFO: found job 6561 in provider local
2024-03-15 10:23:32 proc_lsst.multi:201 [INFO]  cancelling 6561 on provider local
proc_lsst.multi INFO: cancelling 6561 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 6561
2024-03-15 10:23:33 proc_lsst.multi:146 [INFO]  found job 17064200 in provider astro
proc_lsst.multi INFO: found job 17064200 in provider astro
2024-03-15 10:23:33 proc_lsst.multi:201 [INFO]  cancelling 17064200 on provider astro
proc_lsst.multi INFO: cancelling 17064200 on provider astro
2024-03-15 10:23:33 proc_lsst.multi:146 [INFO]  found job 17064219 in provider escience_a40
proc_lsst.multi INFO: found job 17064219 in provider escience_a40
2024-03-15 10:23:33 proc_lsst.multi:201 [INFO]  cancelling 17064219 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064219 on provider escience_a40
2024-03-15 10:23:33 proc_lsst.multi:146 [INFO]  found job 17064191 in provider astro
proc_lsst.multi INFO: found job 17064191 in provider astro
2024-03-15 10:23:33 proc_lsst.multi:201 [INFO]  cancelling 17064191 on provider astro
proc_lsst.multi INFO: cancelling 17064191 on provider astro
2024-03-15 10:23:33 proc_lsst.multi:146 [INFO]  found job 17064209 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064209 in provider escience_rtx6k
2024-03-15 10:23:33 proc_lsst.multi:201 [INFO]  cancelling 17064209 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064209 on provider escience_rtx6k
2024-03-15 10:23:33 proc_lsst.multi:146 [INFO]  found job 17064170 in provider astro
proc_lsst.multi INFO: found job 17064170 in provider astro
2024-03-15 10:23:33 proc_lsst.multi:201 [INFO]  cancelling 17064170 on provider astro
proc_lsst.multi INFO: cancelling 17064170 on provider astro
2024-03-15 10:23:33 proc_lsst.multi:146 [INFO]  found job 17064234 in provider escience_a40
proc_lsst.multi INFO: found job 17064234 in provider escience_a40
2024-03-15 10:23:33 proc_lsst.multi:201 [INFO]  cancelling 17064234 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064234 on provider escience_a40
2024-03-15 10:23:34 proc_lsst.multi:146 [INFO]  found job 17064211 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064211 in provider escience_rtx6k
2024-03-15 10:23:34 proc_lsst.multi:201 [INFO]  cancelling 17064211 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064211 on provider escience_rtx6k
2024-03-15 10:23:34 proc_lsst.multi:146 [INFO]  found job 17064206 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064206 in provider escience_rtx6k
2024-03-15 10:23:34 proc_lsst.multi:201 [INFO]  cancelling 17064206 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064206 on provider escience_rtx6k
2024-03-15 10:23:34 proc_lsst.multi:146 [INFO]  found job 17064228 in provider escience_a40
proc_lsst.multi INFO: found job 17064228 in provider escience_a40
2024-03-15 10:23:34 proc_lsst.multi:201 [INFO]  cancelling 17064228 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064228 on provider escience_a40
2024-03-15 10:23:34 proc_lsst.multi:146 [INFO]  found job 17064197 in provider astro
proc_lsst.multi INFO: found job 17064197 in provider astro
2024-03-15 10:23:34 proc_lsst.multi:201 [INFO]  cancelling 17064197 on provider astro
proc_lsst.multi INFO: cancelling 17064197 on provider astro
2024-03-15 10:23:34 proc_lsst.multi:146 [INFO]  found job 17064195 in provider astro
proc_lsst.multi INFO: found job 17064195 in provider astro
2024-03-15 10:23:34 proc_lsst.multi:201 [INFO]  cancelling 17064195 on provider astro
proc_lsst.multi INFO: cancelling 17064195 on provider astro
2024-03-15 10:23:34 proc_lsst.multi:146 [INFO]  found job 17064203 in provider astro
proc_lsst.multi INFO: found job 17064203 in provider astro
2024-03-15 10:23:34 proc_lsst.multi:201 [INFO]  cancelling 17064203 on provider astro
proc_lsst.multi INFO: cancelling 17064203 on provider astro
2024-03-15 10:23:34 proc_lsst.multi:146 [INFO]  found job 17064224 in provider escience_a40
proc_lsst.multi INFO: found job 17064224 in provider escience_a40
2024-03-15 10:23:34 proc_lsst.multi:201 [INFO]  cancelling 17064224 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064224 on provider escience_a40
2024-03-15 10:23:35 proc_lsst.multi:146 [INFO]  found job 17064196 in provider astro
proc_lsst.multi INFO: found job 17064196 in provider astro
2024-03-15 10:23:35 proc_lsst.multi:201 [INFO]  cancelling 17064196 on provider astro
proc_lsst.multi INFO: cancelling 17064196 on provider astro
2024-03-15 10:23:35 proc_lsst.multi:146 [INFO]  found job 17064194 in provider astro
proc_lsst.multi INFO: found job 17064194 in provider astro
2024-03-15 10:23:35 proc_lsst.multi:201 [INFO]  cancelling 17064194 on provider astro
proc_lsst.multi INFO: cancelling 17064194 on provider astro
2024-03-15 10:23:35 proc_lsst.multi:146 [INFO]  found job 17064202 in provider astro
proc_lsst.multi INFO: found job 17064202 in provider astro
2024-03-15 10:23:35 proc_lsst.multi:201 [INFO]  cancelling 17064202 on provider astro
proc_lsst.multi INFO: cancelling 17064202 on provider astro
2024-03-15 10:23:35 proc_lsst.multi:146 [INFO]  found job 17064220 in provider escience_a40
proc_lsst.multi INFO: found job 17064220 in provider escience_a40
2024-03-15 10:23:35 proc_lsst.multi:201 [INFO]  cancelling 17064220 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064220 on provider escience_a40
2024-03-15 10:23:35 proc_lsst.multi:146 [INFO]  found job 17064213 in provider escience_a40
proc_lsst.multi INFO: found job 17064213 in provider escience_a40
2024-03-15 10:23:35 proc_lsst.multi:201 [INFO]  cancelling 17064213 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064213 on provider escience_a40
2024-03-15 10:23:35 proc_lsst.multi:146 [INFO]  found job 17064178 in provider astro
proc_lsst.multi INFO: found job 17064178 in provider astro
2024-03-15 10:23:35 proc_lsst.multi:201 [INFO]  cancelling 17064178 on provider astro
proc_lsst.multi INFO: cancelling 17064178 on provider astro
2024-03-15 10:23:36 proc_lsst.multi:146 [INFO]  found job 17064240 in provider escience_a40
proc_lsst.multi INFO: found job 17064240 in provider escience_a40
2024-03-15 10:23:36 proc_lsst.multi:201 [INFO]  cancelling 17064240 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064240 on provider escience_a40
2024-03-15 10:23:36 proc_lsst.multi:146 [INFO]  found job 17064218 in provider escience_a40
proc_lsst.multi INFO: found job 17064218 in provider escience_a40
2024-03-15 10:23:36 proc_lsst.multi:201 [INFO]  cancelling 17064218 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064218 on provider escience_a40
2024-03-15 10:23:36 proc_lsst.multi:146 [INFO]  found job 17064189 in provider astro
proc_lsst.multi INFO: found job 17064189 in provider astro
2024-03-15 10:23:36 proc_lsst.multi:201 [INFO]  cancelling 17064189 on provider astro
proc_lsst.multi INFO: cancelling 17064189 on provider astro
2024-03-15 10:23:36 proc_lsst.multi:146 [INFO]  found job 17064186 in provider astro
proc_lsst.multi INFO: found job 17064186 in provider astro
2024-03-15 10:23:36 proc_lsst.multi:201 [INFO]  cancelling 17064186 on provider astro
proc_lsst.multi INFO: cancelling 17064186 on provider astro
2024-03-15 10:23:36 proc_lsst.multi:146 [INFO]  found job 17064192 in provider astro
proc_lsst.multi INFO: found job 17064192 in provider astro
2024-03-15 10:23:36 proc_lsst.multi:201 [INFO]  cancelling 17064192 on provider astro
proc_lsst.multi INFO: cancelling 17064192 on provider astro
2024-03-15 10:23:36 proc_lsst.multi:146 [INFO]  found job 17064210 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064210 in provider escience_rtx6k
2024-03-15 10:23:36 proc_lsst.multi:201 [INFO]  cancelling 17064210 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064210 on provider escience_rtx6k
2024-03-15 10:23:37 proc_lsst.multi:146 [INFO]  found job 17064187 in provider astro
proc_lsst.multi INFO: found job 17064187 in provider astro
2024-03-15 10:23:37 proc_lsst.multi:201 [INFO]  cancelling 17064187 on provider astro
proc_lsst.multi INFO: cancelling 17064187 on provider astro
2024-03-15 10:23:37 proc_lsst.multi:146 [INFO]  found job 17064185 in provider astro
proc_lsst.multi INFO: found job 17064185 in provider astro
2024-03-15 10:23:37 proc_lsst.multi:201 [INFO]  cancelling 17064185 on provider astro
proc_lsst.multi INFO: cancelling 17064185 on provider astro
2024-03-15 10:23:37 proc_lsst.multi:146 [INFO]  found job 17064184 in provider astro
proc_lsst.multi INFO: found job 17064184 in provider astro
2024-03-15 10:23:37 proc_lsst.multi:201 [INFO]  cancelling 17064184 on provider astro
proc_lsst.multi INFO: cancelling 17064184 on provider astro
2024-03-15 10:23:37 proc_lsst.multi:146 [INFO]  found job 17064177 in provider astro
proc_lsst.multi INFO: found job 17064177 in provider astro
2024-03-15 10:23:37 proc_lsst.multi:201 [INFO]  cancelling 17064177 on provider astro
proc_lsst.multi INFO: cancelling 17064177 on provider astro
2024-03-15 10:23:37 proc_lsst.multi:146 [INFO]  found job 17064172 in provider astro
proc_lsst.multi INFO: found job 17064172 in provider astro
2024-03-15 10:23:37 proc_lsst.multi:201 [INFO]  cancelling 17064172 on provider astro
proc_lsst.multi INFO: cancelling 17064172 on provider astro
2024-03-15 10:23:38 proc_lsst.multi:146 [INFO]  found job 17064188 in provider astro
proc_lsst.multi INFO: found job 17064188 in provider astro
2024-03-15 10:23:38 proc_lsst.multi:201 [INFO]  cancelling 17064188 on provider astro
proc_lsst.multi INFO: cancelling 17064188 on provider astro
2024-03-15 10:23:38 proc_lsst.multi:146 [INFO]  found job 17064181 in provider astro
proc_lsst.multi INFO: found job 17064181 in provider astro
2024-03-15 10:23:38 proc_lsst.multi:201 [INFO]  cancelling 17064181 on provider astro
proc_lsst.multi INFO: cancelling 17064181 on provider astro
2024-03-15 10:23:38 proc_lsst.multi:146 [INFO]  found job 17064176 in provider astro
proc_lsst.multi INFO: found job 17064176 in provider astro
2024-03-15 10:23:38 proc_lsst.multi:201 [INFO]  cancelling 17064176 on provider astro
proc_lsst.multi INFO: cancelling 17064176 on provider astro
2024-03-15 10:23:38 proc_lsst.multi:146 [INFO]  found job 17064245 in provider escience_a40
proc_lsst.multi INFO: found job 17064245 in provider escience_a40
2024-03-15 10:23:38 proc_lsst.multi:201 [INFO]  cancelling 17064245 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064245 on provider escience_a40
2024-03-15 10:23:38 proc_lsst.multi:146 [INFO]  found job 17064239 in provider escience_a40
proc_lsst.multi INFO: found job 17064239 in provider escience_a40
2024-03-15 10:23:38 proc_lsst.multi:201 [INFO]  cancelling 17064239 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064239 on provider escience_a40
2024-03-15 10:23:38 proc_lsst.multi:146 [INFO]  found job 17064175 in provider astro
proc_lsst.multi INFO: found job 17064175 in provider astro
2024-03-15 10:23:38 proc_lsst.multi:201 [INFO]  cancelling 17064175 on provider astro
proc_lsst.multi INFO: cancelling 17064175 on provider astro
2024-03-15 10:23:38 proc_lsst.multi:146 [INFO]  found job 17064174 in provider astro
proc_lsst.multi INFO: found job 17064174 in provider astro
2024-03-15 10:23:38 proc_lsst.multi:201 [INFO]  cancelling 17064174 on provider astro
proc_lsst.multi INFO: cancelling 17064174 on provider astro
2024-03-15 10:23:39 proc_lsst.multi:146 [INFO]  found job 17064182 in provider astro
proc_lsst.multi INFO: found job 17064182 in provider astro
2024-03-15 10:23:39 proc_lsst.multi:201 [INFO]  cancelling 17064182 on provider astro
proc_lsst.multi INFO: cancelling 17064182 on provider astro
2024-03-15 10:23:39 proc_lsst.multi:146 [INFO]  found job 17064214 in provider escience_a40
proc_lsst.multi INFO: found job 17064214 in provider escience_a40
2024-03-15 10:23:39 proc_lsst.multi:201 [INFO]  cancelling 17064214 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064214 on provider escience_a40
2024-03-15 10:23:39 proc_lsst.multi:146 [INFO]  found job 17064242 in provider escience_a40
proc_lsst.multi INFO: found job 17064242 in provider escience_a40
2024-03-15 10:23:39 proc_lsst.multi:201 [INFO]  cancelling 17064242 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064242 on provider escience_a40
2024-03-15 10:23:39 proc_lsst.multi:146 [INFO]  found job 17064204 in provider astro
proc_lsst.multi INFO: found job 17064204 in provider astro
2024-03-15 10:23:39 proc_lsst.multi:201 [INFO]  cancelling 17064204 on provider astro
proc_lsst.multi INFO: cancelling 17064204 on provider astro
2024-03-15 10:23:39 proc_lsst.multi:146 [INFO]  found job 17064217 in provider escience_a40
proc_lsst.multi INFO: found job 17064217 in provider escience_a40
2024-03-15 10:23:39 proc_lsst.multi:201 [INFO]  cancelling 17064217 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064217 on provider escience_a40
2024-03-15 10:23:39 proc_lsst.multi:146 [INFO]  found job 17064241 in provider escience_a40
proc_lsst.multi INFO: found job 17064241 in provider escience_a40
2024-03-15 10:23:39 proc_lsst.multi:201 [INFO]  cancelling 17064241 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064241 on provider escience_a40
parsl.dataflow.dflow INFO: Shutting down executor multi
2024-03-15 10:23:39 proc_lsst.multi:40 [INFO]  Cancelling all provider resources
proc_lsst.multi INFO: Cancelling all provider resources
2024-03-15 10:23:39 proc_lsst.multi:47 [INFO]  new jobs since last cancel ['17064259', '17064262', '17064263', '17064264', '17064265', '17064267', '17064268', '17064273', '17064279', '17064280', '17064281', '17064282', '17064285', '17064286', '17064287', '17064289', '17064290', '17064291', '17064292', '17064294', '17064296', '17064297', '17064298', '17064299', '17064300', '17064301', '17064302', '17064303', '17064304', '17064305', '17064306', '17064307', '17064308', '17064309', '17064310', '17064311', '17064312', '17064169', '17064170', '17064171', '17064172', '17064173', '17064174', '17064175', '17064176', '17064177', '17064178', '17064180', '17064181', '17064182', '17064183', '17064184', '17064185', '17064186', '17064187', '17064188', '17064189', '17064191', '17064192', '17064194', '17064195', '17064196', '17064197', '17064200', '17064202', '17064203', '17064204', '17064213', '17064214', '17064215', '17064217', '17064218', '17064219', '17064220', '17064221', '17064222', '17064223', '17064224', '17064228', '17064233', '17064234', '17064239', '17064240', '17064241', '17064242', '17064243', '17064245', '17064206', '17064208', '17064209', '17064210', '17064211', '6561']
proc_lsst.multi INFO: new jobs since last cancel ['17064259', '17064262', '17064263', '17064264', '17064265', '17064267', '17064268', '17064273', '17064279', '17064280', '17064281', '17064282', '17064285', '17064286', '17064287', '17064289', '17064290', '17064291', '17064292', '17064294', '17064296', '17064297', '17064298', '17064299', '17064300', '17064301', '17064302', '17064303', '17064304', '17064305', '17064306', '17064307', '17064308', '17064309', '17064310', '17064311', '17064312', '17064169', '17064170', '17064171', '17064172', '17064173', '17064174', '17064175', '17064176', '17064177', '17064178', '17064180', '17064181', '17064182', '17064183', '17064184', '17064185', '17064186', '17064187', '17064188', '17064189', '17064191', '17064192', '17064194', '17064195', '17064196', '17064197', '17064200', '17064202', '17064203', '17064204', '17064213', '17064214', '17064215', '17064217', '17064218', '17064219', '17064220', '17064221', '17064222', '17064223', '17064224', '17064228', '17064233', '17064234', '17064239', '17064240', '17064241', '17064242', '17064243', '17064245', '17064206', '17064208', '17064209', '17064210', '17064211', '6561']
2024-03-15 10:23:39 proc_lsst.multi:146 [INFO]  found job 17064259 in provider checkpoint
proc_lsst.multi INFO: found job 17064259 in provider checkpoint
2024-03-15 10:23:39 proc_lsst.multi:201 [INFO]  cancelling 17064259 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064259 on provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:146 [INFO]  found job 17064262 in provider checkpoint
proc_lsst.multi INFO: found job 17064262 in provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:201 [INFO]  cancelling 17064262 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064262 on provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:146 [INFO]  found job 17064263 in provider checkpoint
proc_lsst.multi INFO: found job 17064263 in provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:201 [INFO]  cancelling 17064263 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064263 on provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:146 [INFO]  found job 17064264 in provider checkpoint
proc_lsst.multi INFO: found job 17064264 in provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:201 [INFO]  cancelling 17064264 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064264 on provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:146 [INFO]  found job 17064265 in provider checkpoint
proc_lsst.multi INFO: found job 17064265 in provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:201 [INFO]  cancelling 17064265 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064265 on provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:146 [INFO]  found job 17064267 in provider checkpoint
proc_lsst.multi INFO: found job 17064267 in provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:201 [INFO]  cancelling 17064267 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064267 on provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:146 [INFO]  found job 17064268 in provider checkpoint
proc_lsst.multi INFO: found job 17064268 in provider checkpoint
2024-03-15 10:23:40 proc_lsst.multi:201 [INFO]  cancelling 17064268 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064268 on provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:146 [INFO]  found job 17064273 in provider checkpoint
proc_lsst.multi INFO: found job 17064273 in provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:201 [INFO]  cancelling 17064273 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064273 on provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:146 [INFO]  found job 17064279 in provider checkpoint
proc_lsst.multi INFO: found job 17064279 in provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:201 [INFO]  cancelling 17064279 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064279 on provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:146 [INFO]  found job 17064280 in provider checkpoint
proc_lsst.multi INFO: found job 17064280 in provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:201 [INFO]  cancelling 17064280 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064280 on provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:146 [INFO]  found job 17064281 in provider checkpoint
proc_lsst.multi INFO: found job 17064281 in provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:201 [INFO]  cancelling 17064281 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064281 on provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:146 [INFO]  found job 17064282 in provider checkpoint
proc_lsst.multi INFO: found job 17064282 in provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:201 [INFO]  cancelling 17064282 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064282 on provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:146 [INFO]  found job 17064285 in provider checkpoint
proc_lsst.multi INFO: found job 17064285 in provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:201 [INFO]  cancelling 17064285 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064285 on provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:146 [INFO]  found job 17064286 in provider checkpoint
proc_lsst.multi INFO: found job 17064286 in provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:201 [INFO]  cancelling 17064286 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064286 on provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:146 [INFO]  found job 17064287 in provider checkpoint
proc_lsst.multi INFO: found job 17064287 in provider checkpoint
2024-03-15 10:23:41 proc_lsst.multi:201 [INFO]  cancelling 17064287 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064287 on provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:146 [INFO]  found job 17064289 in provider checkpoint
proc_lsst.multi INFO: found job 17064289 in provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:201 [INFO]  cancelling 17064289 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064289 on provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:146 [INFO]  found job 17064290 in provider checkpoint
proc_lsst.multi INFO: found job 17064290 in provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:201 [INFO]  cancelling 17064290 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064290 on provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:146 [INFO]  found job 17064291 in provider checkpoint
proc_lsst.multi INFO: found job 17064291 in provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:201 [INFO]  cancelling 17064291 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064291 on provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:146 [INFO]  found job 17064292 in provider checkpoint
proc_lsst.multi INFO: found job 17064292 in provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:201 [INFO]  cancelling 17064292 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064292 on provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:146 [INFO]  found job 17064294 in provider checkpoint
proc_lsst.multi INFO: found job 17064294 in provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:201 [INFO]  cancelling 17064294 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064294 on provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:146 [INFO]  found job 17064296 in provider checkpoint
proc_lsst.multi INFO: found job 17064296 in provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:201 [INFO]  cancelling 17064296 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064296 on provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:146 [INFO]  found job 17064297 in provider checkpoint
proc_lsst.multi INFO: found job 17064297 in provider checkpoint
2024-03-15 10:23:42 proc_lsst.multi:201 [INFO]  cancelling 17064297 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064297 on provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:146 [INFO]  found job 17064298 in provider checkpoint
proc_lsst.multi INFO: found job 17064298 in provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:201 [INFO]  cancelling 17064298 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064298 on provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:146 [INFO]  found job 17064299 in provider checkpoint
proc_lsst.multi INFO: found job 17064299 in provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:201 [INFO]  cancelling 17064299 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064299 on provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:146 [INFO]  found job 17064300 in provider checkpoint
proc_lsst.multi INFO: found job 17064300 in provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:201 [INFO]  cancelling 17064300 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064300 on provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:146 [INFO]  found job 17064301 in provider checkpoint
proc_lsst.multi INFO: found job 17064301 in provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:201 [INFO]  cancelling 17064301 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064301 on provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:146 [INFO]  found job 17064302 in provider checkpoint
proc_lsst.multi INFO: found job 17064302 in provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:201 [INFO]  cancelling 17064302 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064302 on provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:146 [INFO]  found job 17064303 in provider checkpoint
proc_lsst.multi INFO: found job 17064303 in provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:201 [INFO]  cancelling 17064303 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064303 on provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:146 [INFO]  found job 17064304 in provider checkpoint
proc_lsst.multi INFO: found job 17064304 in provider checkpoint
2024-03-15 10:23:43 proc_lsst.multi:201 [INFO]  cancelling 17064304 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064304 on provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:146 [INFO]  found job 17064305 in provider checkpoint
proc_lsst.multi INFO: found job 17064305 in provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:201 [INFO]  cancelling 17064305 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064305 on provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:146 [INFO]  found job 17064306 in provider checkpoint
proc_lsst.multi INFO: found job 17064306 in provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:201 [INFO]  cancelling 17064306 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064306 on provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:146 [INFO]  found job 17064307 in provider checkpoint
proc_lsst.multi INFO: found job 17064307 in provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:201 [INFO]  cancelling 17064307 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064307 on provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:146 [INFO]  found job 17064308 in provider checkpoint
proc_lsst.multi INFO: found job 17064308 in provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:201 [INFO]  cancelling 17064308 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064308 on provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:146 [INFO]  found job 17064309 in provider checkpoint
proc_lsst.multi INFO: found job 17064309 in provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:201 [INFO]  cancelling 17064309 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064309 on provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:146 [INFO]  found job 17064310 in provider checkpoint
proc_lsst.multi INFO: found job 17064310 in provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:201 [INFO]  cancelling 17064310 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064310 on provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:146 [INFO]  found job 17064311 in provider checkpoint
proc_lsst.multi INFO: found job 17064311 in provider checkpoint
2024-03-15 10:23:44 proc_lsst.multi:201 [INFO]  cancelling 17064311 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064311 on provider checkpoint
2024-03-15 10:23:45 proc_lsst.multi:146 [INFO]  found job 17064312 in provider checkpoint
proc_lsst.multi INFO: found job 17064312 in provider checkpoint
2024-03-15 10:23:45 proc_lsst.multi:201 [INFO]  cancelling 17064312 on provider checkpoint
proc_lsst.multi INFO: cancelling 17064312 on provider checkpoint
2024-03-15 10:23:45 proc_lsst.multi:146 [INFO]  found job 17064169 in provider astro
proc_lsst.multi INFO: found job 17064169 in provider astro
2024-03-15 10:23:45 proc_lsst.multi:201 [INFO]  cancelling 17064169 on provider astro
proc_lsst.multi INFO: cancelling 17064169 on provider astro
2024-03-15 10:23:45 proc_lsst.multi:146 [INFO]  found job 17064170 in provider astro
proc_lsst.multi INFO: found job 17064170 in provider astro
2024-03-15 10:23:45 proc_lsst.multi:201 [INFO]  cancelling 17064170 on provider astro
proc_lsst.multi INFO: cancelling 17064170 on provider astro
2024-03-15 10:23:45 proc_lsst.multi:146 [INFO]  found job 17064171 in provider astro
proc_lsst.multi INFO: found job 17064171 in provider astro
2024-03-15 10:23:45 proc_lsst.multi:201 [INFO]  cancelling 17064171 on provider astro
proc_lsst.multi INFO: cancelling 17064171 on provider astro
2024-03-15 10:23:45 proc_lsst.multi:146 [INFO]  found job 17064172 in provider astro
proc_lsst.multi INFO: found job 17064172 in provider astro
2024-03-15 10:23:45 proc_lsst.multi:201 [INFO]  cancelling 17064172 on provider astro
proc_lsst.multi INFO: cancelling 17064172 on provider astro
2024-03-15 10:23:45 proc_lsst.multi:146 [INFO]  found job 17064173 in provider astro
proc_lsst.multi INFO: found job 17064173 in provider astro
2024-03-15 10:23:45 proc_lsst.multi:201 [INFO]  cancelling 17064173 on provider astro
proc_lsst.multi INFO: cancelling 17064173 on provider astro
2024-03-15 10:23:45 proc_lsst.multi:146 [INFO]  found job 17064174 in provider astro
proc_lsst.multi INFO: found job 17064174 in provider astro
2024-03-15 10:23:45 proc_lsst.multi:201 [INFO]  cancelling 17064174 on provider astro
proc_lsst.multi INFO: cancelling 17064174 on provider astro
2024-03-15 10:23:45 proc_lsst.multi:146 [INFO]  found job 17064175 in provider astro
proc_lsst.multi INFO: found job 17064175 in provider astro
2024-03-15 10:23:45 proc_lsst.multi:201 [INFO]  cancelling 17064175 on provider astro
proc_lsst.multi INFO: cancelling 17064175 on provider astro
2024-03-15 10:23:46 proc_lsst.multi:146 [INFO]  found job 17064176 in provider astro
proc_lsst.multi INFO: found job 17064176 in provider astro
2024-03-15 10:23:46 proc_lsst.multi:201 [INFO]  cancelling 17064176 on provider astro
proc_lsst.multi INFO: cancelling 17064176 on provider astro
2024-03-15 10:23:46 proc_lsst.multi:146 [INFO]  found job 17064177 in provider astro
proc_lsst.multi INFO: found job 17064177 in provider astro
2024-03-15 10:23:46 proc_lsst.multi:201 [INFO]  cancelling 17064177 on provider astro
proc_lsst.multi INFO: cancelling 17064177 on provider astro
2024-03-15 10:23:46 proc_lsst.multi:146 [INFO]  found job 17064178 in provider astro
proc_lsst.multi INFO: found job 17064178 in provider astro
2024-03-15 10:23:46 proc_lsst.multi:201 [INFO]  cancelling 17064178 on provider astro
proc_lsst.multi INFO: cancelling 17064178 on provider astro
2024-03-15 10:23:46 proc_lsst.multi:146 [INFO]  found job 17064180 in provider astro
proc_lsst.multi INFO: found job 17064180 in provider astro
2024-03-15 10:23:46 proc_lsst.multi:201 [INFO]  cancelling 17064180 on provider astro
proc_lsst.multi INFO: cancelling 17064180 on provider astro
2024-03-15 10:23:46 proc_lsst.multi:146 [INFO]  found job 17064181 in provider astro
proc_lsst.multi INFO: found job 17064181 in provider astro
2024-03-15 10:23:46 proc_lsst.multi:201 [INFO]  cancelling 17064181 on provider astro
proc_lsst.multi INFO: cancelling 17064181 on provider astro
2024-03-15 10:23:46 proc_lsst.multi:146 [INFO]  found job 17064182 in provider astro
proc_lsst.multi INFO: found job 17064182 in provider astro
2024-03-15 10:23:46 proc_lsst.multi:201 [INFO]  cancelling 17064182 on provider astro
proc_lsst.multi INFO: cancelling 17064182 on provider astro
2024-03-15 10:23:46 proc_lsst.multi:146 [INFO]  found job 17064183 in provider astro
proc_lsst.multi INFO: found job 17064183 in provider astro
2024-03-15 10:23:46 proc_lsst.multi:201 [INFO]  cancelling 17064183 on provider astro
proc_lsst.multi INFO: cancelling 17064183 on provider astro
2024-03-15 10:23:47 proc_lsst.multi:146 [INFO]  found job 17064184 in provider astro
proc_lsst.multi INFO: found job 17064184 in provider astro
2024-03-15 10:23:47 proc_lsst.multi:201 [INFO]  cancelling 17064184 on provider astro
proc_lsst.multi INFO: cancelling 17064184 on provider astro
2024-03-15 10:23:47 proc_lsst.multi:146 [INFO]  found job 17064185 in provider astro
proc_lsst.multi INFO: found job 17064185 in provider astro
2024-03-15 10:23:47 proc_lsst.multi:201 [INFO]  cancelling 17064185 on provider astro
proc_lsst.multi INFO: cancelling 17064185 on provider astro
2024-03-15 10:23:47 proc_lsst.multi:146 [INFO]  found job 17064186 in provider astro
proc_lsst.multi INFO: found job 17064186 in provider astro
2024-03-15 10:23:47 proc_lsst.multi:201 [INFO]  cancelling 17064186 on provider astro
proc_lsst.multi INFO: cancelling 17064186 on provider astro
2024-03-15 10:23:47 proc_lsst.multi:146 [INFO]  found job 17064187 in provider astro
proc_lsst.multi INFO: found job 17064187 in provider astro
2024-03-15 10:23:47 proc_lsst.multi:201 [INFO]  cancelling 17064187 on provider astro
proc_lsst.multi INFO: cancelling 17064187 on provider astro
2024-03-15 10:23:47 proc_lsst.multi:146 [INFO]  found job 17064188 in provider astro
proc_lsst.multi INFO: found job 17064188 in provider astro
2024-03-15 10:23:47 proc_lsst.multi:201 [INFO]  cancelling 17064188 on provider astro
proc_lsst.multi INFO: cancelling 17064188 on provider astro
2024-03-15 10:23:47 proc_lsst.multi:146 [INFO]  found job 17064189 in provider astro
proc_lsst.multi INFO: found job 17064189 in provider astro
2024-03-15 10:23:47 proc_lsst.multi:201 [INFO]  cancelling 17064189 on provider astro
proc_lsst.multi INFO: cancelling 17064189 on provider astro
2024-03-15 10:23:47 proc_lsst.multi:146 [INFO]  found job 17064191 in provider astro
proc_lsst.multi INFO: found job 17064191 in provider astro
2024-03-15 10:23:47 proc_lsst.multi:201 [INFO]  cancelling 17064191 on provider astro
proc_lsst.multi INFO: cancelling 17064191 on provider astro
2024-03-15 10:23:47 proc_lsst.multi:146 [INFO]  found job 17064192 in provider astro
proc_lsst.multi INFO: found job 17064192 in provider astro
2024-03-15 10:23:47 proc_lsst.multi:201 [INFO]  cancelling 17064192 on provider astro
proc_lsst.multi INFO: cancelling 17064192 on provider astro
2024-03-15 10:23:48 proc_lsst.multi:146 [INFO]  found job 17064194 in provider astro
proc_lsst.multi INFO: found job 17064194 in provider astro
2024-03-15 10:23:48 proc_lsst.multi:201 [INFO]  cancelling 17064194 on provider astro
proc_lsst.multi INFO: cancelling 17064194 on provider astro
2024-03-15 10:23:48 proc_lsst.multi:146 [INFO]  found job 17064195 in provider astro
proc_lsst.multi INFO: found job 17064195 in provider astro
2024-03-15 10:23:48 proc_lsst.multi:201 [INFO]  cancelling 17064195 on provider astro
proc_lsst.multi INFO: cancelling 17064195 on provider astro
2024-03-15 10:23:48 proc_lsst.multi:146 [INFO]  found job 17064196 in provider astro
proc_lsst.multi INFO: found job 17064196 in provider astro
2024-03-15 10:23:48 proc_lsst.multi:201 [INFO]  cancelling 17064196 on provider astro
proc_lsst.multi INFO: cancelling 17064196 on provider astro
2024-03-15 10:23:48 proc_lsst.multi:146 [INFO]  found job 17064197 in provider astro
proc_lsst.multi INFO: found job 17064197 in provider astro
2024-03-15 10:23:48 proc_lsst.multi:201 [INFO]  cancelling 17064197 on provider astro
proc_lsst.multi INFO: cancelling 17064197 on provider astro
2024-03-15 10:23:49 proc_lsst.multi:146 [INFO]  found job 17064200 in provider astro
proc_lsst.multi INFO: found job 17064200 in provider astro
2024-03-15 10:23:49 proc_lsst.multi:201 [INFO]  cancelling 17064200 on provider astro
proc_lsst.multi INFO: cancelling 17064200 on provider astro
2024-03-15 10:23:49 proc_lsst.multi:146 [INFO]  found job 17064202 in provider astro
proc_lsst.multi INFO: found job 17064202 in provider astro
2024-03-15 10:23:49 proc_lsst.multi:201 [INFO]  cancelling 17064202 on provider astro
proc_lsst.multi INFO: cancelling 17064202 on provider astro
2024-03-15 10:23:49 proc_lsst.multi:146 [INFO]  found job 17064203 in provider astro
proc_lsst.multi INFO: found job 17064203 in provider astro
2024-03-15 10:23:49 proc_lsst.multi:201 [INFO]  cancelling 17064203 on provider astro
proc_lsst.multi INFO: cancelling 17064203 on provider astro
2024-03-15 10:23:49 proc_lsst.multi:146 [INFO]  found job 17064204 in provider astro
proc_lsst.multi INFO: found job 17064204 in provider astro
2024-03-15 10:23:49 proc_lsst.multi:201 [INFO]  cancelling 17064204 on provider astro
proc_lsst.multi INFO: cancelling 17064204 on provider astro
2024-03-15 10:23:49 proc_lsst.multi:146 [INFO]  found job 17064213 in provider escience_a40
proc_lsst.multi INFO: found job 17064213 in provider escience_a40
2024-03-15 10:23:49 proc_lsst.multi:201 [INFO]  cancelling 17064213 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064213 on provider escience_a40
2024-03-15 10:23:49 proc_lsst.multi:146 [INFO]  found job 17064214 in provider escience_a40
proc_lsst.multi INFO: found job 17064214 in provider escience_a40
2024-03-15 10:23:49 proc_lsst.multi:201 [INFO]  cancelling 17064214 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064214 on provider escience_a40
2024-03-15 10:23:49 proc_lsst.multi:146 [INFO]  found job 17064215 in provider escience_a40
proc_lsst.multi INFO: found job 17064215 in provider escience_a40
2024-03-15 10:23:49 proc_lsst.multi:201 [INFO]  cancelling 17064215 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064215 on provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:146 [INFO]  found job 17064217 in provider escience_a40
proc_lsst.multi INFO: found job 17064217 in provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:201 [INFO]  cancelling 17064217 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064217 on provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:146 [INFO]  found job 17064218 in provider escience_a40
proc_lsst.multi INFO: found job 17064218 in provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:201 [INFO]  cancelling 17064218 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064218 on provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:146 [INFO]  found job 17064219 in provider escience_a40
proc_lsst.multi INFO: found job 17064219 in provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:201 [INFO]  cancelling 17064219 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064219 on provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:146 [INFO]  found job 17064220 in provider escience_a40
proc_lsst.multi INFO: found job 17064220 in provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:201 [INFO]  cancelling 17064220 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064220 on provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:146 [INFO]  found job 17064221 in provider escience_a40
proc_lsst.multi INFO: found job 17064221 in provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:201 [INFO]  cancelling 17064221 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064221 on provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:146 [INFO]  found job 17064222 in provider escience_a40
proc_lsst.multi INFO: found job 17064222 in provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:201 [INFO]  cancelling 17064222 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064222 on provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:146 [INFO]  found job 17064223 in provider escience_a40
proc_lsst.multi INFO: found job 17064223 in provider escience_a40
2024-03-15 10:23:50 proc_lsst.multi:201 [INFO]  cancelling 17064223 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064223 on provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:146 [INFO]  found job 17064224 in provider escience_a40
proc_lsst.multi INFO: found job 17064224 in provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:201 [INFO]  cancelling 17064224 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064224 on provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:146 [INFO]  found job 17064228 in provider escience_a40
proc_lsst.multi INFO: found job 17064228 in provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:201 [INFO]  cancelling 17064228 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064228 on provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:146 [INFO]  found job 17064233 in provider escience_a40
proc_lsst.multi INFO: found job 17064233 in provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:201 [INFO]  cancelling 17064233 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064233 on provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:146 [INFO]  found job 17064234 in provider escience_a40
proc_lsst.multi INFO: found job 17064234 in provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:201 [INFO]  cancelling 17064234 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064234 on provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:146 [INFO]  found job 17064239 in provider escience_a40
proc_lsst.multi INFO: found job 17064239 in provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:201 [INFO]  cancelling 17064239 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064239 on provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:146 [INFO]  found job 17064240 in provider escience_a40
proc_lsst.multi INFO: found job 17064240 in provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:201 [INFO]  cancelling 17064240 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064240 on provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:146 [INFO]  found job 17064241 in provider escience_a40
proc_lsst.multi INFO: found job 17064241 in provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:201 [INFO]  cancelling 17064241 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064241 on provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:146 [INFO]  found job 17064242 in provider escience_a40
proc_lsst.multi INFO: found job 17064242 in provider escience_a40
2024-03-15 10:23:51 proc_lsst.multi:201 [INFO]  cancelling 17064242 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064242 on provider escience_a40
2024-03-15 10:23:52 proc_lsst.multi:146 [INFO]  found job 17064243 in provider escience_a40
proc_lsst.multi INFO: found job 17064243 in provider escience_a40
2024-03-15 10:23:52 proc_lsst.multi:201 [INFO]  cancelling 17064243 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064243 on provider escience_a40
2024-03-15 10:23:52 proc_lsst.multi:146 [INFO]  found job 17064245 in provider escience_a40
proc_lsst.multi INFO: found job 17064245 in provider escience_a40
2024-03-15 10:23:52 proc_lsst.multi:201 [INFO]  cancelling 17064245 on provider escience_a40
proc_lsst.multi INFO: cancelling 17064245 on provider escience_a40
2024-03-15 10:23:52 proc_lsst.multi:146 [INFO]  found job 17064206 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064206 in provider escience_rtx6k
2024-03-15 10:23:52 proc_lsst.multi:201 [INFO]  cancelling 17064206 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064206 on provider escience_rtx6k
2024-03-15 10:23:52 proc_lsst.multi:146 [INFO]  found job 17064208 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064208 in provider escience_rtx6k
2024-03-15 10:23:52 proc_lsst.multi:201 [INFO]  cancelling 17064208 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064208 on provider escience_rtx6k
2024-03-15 10:23:52 proc_lsst.multi:146 [INFO]  found job 17064209 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064209 in provider escience_rtx6k
2024-03-15 10:23:52 proc_lsst.multi:201 [INFO]  cancelling 17064209 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064209 on provider escience_rtx6k
2024-03-15 10:23:52 proc_lsst.multi:146 [INFO]  found job 17064210 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064210 in provider escience_rtx6k
2024-03-15 10:23:52 proc_lsst.multi:201 [INFO]  cancelling 17064210 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064210 on provider escience_rtx6k
2024-03-15 10:23:52 proc_lsst.multi:146 [INFO]  found job 17064211 in provider escience_rtx6k
proc_lsst.multi INFO: found job 17064211 in provider escience_rtx6k
2024-03-15 10:23:52 proc_lsst.multi:201 [INFO]  cancelling 17064211 on provider escience_rtx6k
proc_lsst.multi INFO: cancelling 17064211 on provider escience_rtx6k
2024-03-15 10:23:53 proc_lsst.multi:146 [INFO]  found job 6561 in provider local
proc_lsst.multi INFO: found job 6561 in provider local
2024-03-15 10:23:53 proc_lsst.multi:201 [INFO]  cancelling 6561 on provider local
proc_lsst.multi INFO: cancelling 6561 on provider local
parsl.providers.local.local DEBUG: Terminating job/proc_id: 6561
parsl.providers.local.local WARNING: Failed to kill PID: 6561 and child processes on local
2024-03-15 10:23:54 proc_lsst.multi:50 [INFO]  no new jobs since last cancel, resuming executor shutdown
proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown
parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown
parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt
parsl.dataflow.dflow INFO: Shut down executor multi
parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal
parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete
parsl.executors.threads DEBUG: Done with executor shutdown
parsl.dataflow.dflow INFO: Shut down executor _parsl_internal
parsl.dataflow.dflow INFO: Terminated executors
parsl.dataflow.dflow INFO: DFK cleanup complete
parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread
lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 161.4705 seconds
lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210510_A1h_science#step0_20240315T102059Z' submitted for execution with id 'None'
lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 161.4769 seconds; current memory usage: 0.279 Gibyte, delta: 0.005 Gibyte, peak delta: 0.007 Gibyte
lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 176.7411 seconds; current memory usage: 0.279 Gibyte, delta: 0.142 Gibyte, peak delta: 0.144 Gibyte
lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.281 Gibyte (main), 0.281 Gibyte (largest child process)
Run Id: None
Run Name: DEEP_20210510_A1h_science#step0_20240315T102059Z
parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up

real	2m58.796s
user	0m34.885s
sys	0m16.629s